Kubernetes internals¶

The details behind Labs on Kubernetes: what the server creates, whose identity each call uses, what clients see, and how nodes are reached.

Users and their rights¶

Provisioning (the server's identity). On the first deploy of a lab the server creates its namespace (c9s-<user>-<lab>, or c9s-shared-<lab> for a topology under @shared/) with the labels containerlab.dev/runtime=clabernetes, clab-owner=<user>, clab-lab=<lab>, app.kubernetes.io/managed-by=clab-api-server and pod-security.kubernetes.io/enforce=privileged, and in it:

• a RoleBinding of clabernetes-lab-editor to clab:<user> (or the group clab:shared-<group>); the role also grants get on the namespace object, which containerlab reads before it deploys into it, and bound in the namespace that is the lab's namespace only,
• the ResourceQuota from CLAB_KUBE_LAB_QUOTA, and a copy of the image pull secret, if set.

Lab calls (the user's identity). Deploy, apply, destroy, inspect, exec, save, start/stop/restart, events, logs, terminals, port-forwards and stats run through the containerlab library as the user, with Kubernetes impersonation. The cluster's audit log names the user, and a user has the same rights through the server as with kubectl.

Listing. The server lists the namespaces labelled clab-owner=<user> (and the shared ones of the user's groups) with its own identity, then inspects each as the user. Superusers inspect every namespace as clab:admins. The events stream watches each of the user's lab namespaces as the user.

Destroy. A plain destroy keeps the node storage (the saved configuration) and therefore the namespace; the next deploy reuses both. The server deletes the namespace once containerlab removed the lab and no claims are left, and always on cleanup=true. It only deletes namespaces it created.

Names. Lab names are unique per user, not per server: two users may both have a lab demo. A superuser addressing a lab name that several other users have gets 409.

The server's own checks (clab-owner, superuser, shared workspace) stay in place, so a request fails closed if either the server or Kubernetes refuses it.

What users can edit. The lab editor role lets users update their Topology, which carries the lab's clab-owner label and the topology file containerlab deployed from (annotation containerlab.dev/topo-file). The server reads, applies and purges that file, so it does not take either from the Topology: a lab's owner is the one of its namespace, and its topology file (labPath/absLabPath in inspect) is kept only when it lies in the owner's workspace, or for a shared lab in the shared workspace under a directory of the lab's group. Otherwise inspect reports k8s://<namespace>/topologies/<lab>, as for a lab without a recorded file, and file operations that need the topology answer 404. The events stream corrects the clab-owner and clab-topo-file attributes of container events the same way, so they always match inspect; an event whose records the server cannot check (the cluster does not answer) is left out.

The server's own ClusterRole, clab-api-server-provisioner:

It has no rights on clabernetes resources or Pods itself. The server can impersonate any user name, so treat its credentials like a cluster administrator's.

Startup checks¶

Before it serves requests the server:

1. runs a SelfSubjectAccessReview for every permission of clab-api-server-provisioner and lists every one that is denied,
2. runs containerlab's clabernetes release check: the published OpenAPI schema of nodes.c9s.run must have spec.runState (any authenticated identity may read it), and
3. checks that the ClusterRole clabernetes-lab-editor exists.

If any of them fails, it exits with a message that names what is missing.

What clients see¶

GET /health reports the runtime:

{
  "status": "healthy",
  "runtime": "c9s",
  "namespaceMode": "per-lab",
  "features": ["validate", "graph-mermaid", "graph-dot", "exec-filters", "lab-lifecycle-node-filter",
               "save-copy", "inspect-details", "node-stats", "log-stream", "terminal-resume",
               "node-logs-runtime", "interface-state", "c9s-runtime", "lab-host-info",
               "capture-stream", "wireshark-vnc", "lab-lifecycle-timeout"]
}

namespaceMode is always per-lab: every lab has its own namespace.

GET /api/v1/lab-host (any signed-in user) names the cluster and the deploy defaults the user's labs get:

{
  "runtime": "c9s",
  "defaultTimeout": "10m",
  "capture": {
    "wiresharkVnc": false,
    "wiresharkImage": "ghcr.io/srl-labs/wireshark-vnc-docker:latest",
    "wiresharkPullPolicy": "always"
  },
  "c9s": {
    "context": "c9s-provisioner",
    "cluster": "c9s",
    "namespacePattern": "c9s-alice-{lab}",
    "sharedNamespacePattern": "c9s-shared-{lab}",
    "exposeType": "None",
    "persistence": true,
    "imagePullSecret": "regcred",
    "labQuota": false,
    "noTopologyCR": false,
    "metrics": true
  }
}

context and cluster are names from the server's kubeconfig ("" and in-cluster with the Pod's ServiceAccount); the cluster's address, credentials and file paths are never reported. namespacePattern is CLAB_KUBE_NAMESPACE_TEMPLATE rendered for the signed-in user with {lab} left in, imagePullSecret the secret's name without its namespace, labQuota whether CLAB_KUBE_LAB_QUOTA is set, and metrics whether metrics.k8s.io is served. noTopologyCR is always false: the server deploys every lab with clabernetes' Topology resource, which carries the lab's owner and topology file. defaultTimeout is containerlab's timeout of a deploy, redeploy, apply or destroy that passes no timeout (see Timeouts).

capture describes packet capture with every runtime, with the defaults applied: wiresharkVnc whether the server runs Wireshark VNC sessions for the user's labs (on c9s while Docker answers on the server host, as wireshark-vnc in /health), wiresharkImage and wiresharkPullPolicy their image and when it is pulled (CAPTURE_WIRESHARK_DOCKER_IMAGE, CAPTURE_WIRESHARK_PULL_POLICY: always, missing or never). Docker and Podman servers answer without c9s and add Edgeshark, which runs next to their labs: its packetflix port (CAPTURE_PACKETFLIX_PORT) and the names of the variables in CAPTURE_EDGESHARK_EXTRA_ENV_VARS, never their values:

{
  "runtime": "docker",
  "defaultTimeout": "5m",
  "capture": {
    "wiresharkVnc": true,
    "wiresharkImage": "ghcr.io/srl-labs/wireshark-vnc-docker:latest",
    "wiresharkPullPolicy": "always",
    "edgeshark": { "packetflixPort": 5001, "extraEnvVars": ["HTTP_PROXY", "NO_PROXY"] }
  }
}

Features that need Docker or Podman on the server host are left out on c9s: snapshot-save, deploy-restore, sshx-readers, gotty-options.

Inspect records of c9s nodes carry namespace, pod, container, node_uid, exposed_ports and load_balancer_address; container_id is <namespace>/<lab>/<node> and the node name is <lab>-<node>. Otherwise they read like Docker's:

• state is created, running or exited, and status Created, health: starting, healthy or Exited;
• labPath/absLabPath is the topology file on the server host the lab was deployed from (see What users can edit for when the server reports k8s://<namespace>/topologies/<lab> instead).

The events stream is containerlab's: container events with Docker's actions (create, start, kill, stop, die, destroy, health_status: ...) and attributes (containerlab, clab-node-name, clab-node-kind, clab-owner, clab-topo-file, image, mgmt_ipv4, state, ...), plus namespace, node and pod; interface events for the node's link interfaces under their topology names, eth0 and lo. A user's stream covers the namespaces of the user's labs; a superuser's every namespace.

How the server reaches nodes¶

Lab names must be lowercase DNS labels (letters, digits and dashes, starting with a letter). Other names are refused with the name to use instead.

Timeouts¶

On c9s containerlab waits up to 10 minutes by default for a lab's Pods, as containerlab deploy -r c9s does: a cluster pulls node images and schedules Pods slower than a local Docker. Deploy, redeploy, apply and destroy take an optional timeout query parameter (containerlab's --timeout, a Go duration such as 30m, at most 2h) for labs with large images. The request runs for the timeout plus 5 minutes, and a redeploy for twice the timeout plus 5 minutes, so containerlab's own timeout error reaches the client.

Packet capture¶

On c9s the server captures through containerlab's c9s capture: clabwire, the sidecar in every lab pod, captures the requested lab link interfaces, and containerlab merges them into one pcapng section. Every call runs as the signed-in user, so Kubernetes decides whether the user may capture (pods/exec on the lab's pods, which clabernetes-lab-editor grants). Edgeshark is not used: clabwire only allows the interfaces of the lab's links, so the management interface and c9s0 cannot be captured.

The stream has one interface per target, named <node>:<interface>, before any packet, and packets in timestamp order. Captures longer than clabwire's one-hour limit are chained by containerlab without gaps or duplicates. When the client goes away the capture stops in the pod (clabwire ends on stdin close).

List capturable interfaces¶

GET /api/v1/labs/{lab}/capture/interfaces

Returns the lab link interfaces of the running nodes, with the far end of each link:

[
  {
    "node": "srl1",
    "interface": "e1-1",
    "alias": "ethernet-1/1",
    "peerNode": "srl2",
    "peerInterface": "e1-1"
  }
]

alias appears only when the interface has one.

Stream a capture¶

GET /api/v1/labs/{lab}/capture/stream?target=<node>:<interface>

Returns application/x-pcapng, chunked and flushed as packets arrive. The stream runs until the duration or packet limit is reached, or the client disconnects.

Errors before the first byte are JSON; later ones end the stream.

A target names the node as the topology does (or its sanitized name) and the interface by name or alias, e.g. srl1:ethernet-1/1. The node ends at the first colon.

Wireshark VNC. POST /api/v1/labs/{lab}/capture/wireshark-vnc-sessions takes the node's inspect name (<lab>-<node>) as containerName and an interface (or alias) as interfaceName, and starts one Wireshark VNC container per node on the server host. The server captures the node's requested interfaces as the signed-in user, in one pcapng stream like capture/stream, and writes it into a named pipe in the container (docker exec -i), which the container's Wireshark reads. The container is not attached to an Edgeshark network and does not reach the server. This needs Docker on the server host; without it the routes answer 501 and /health leaves out wireshark-vnc.

The pipe is the CLABWIRE_CAPTURE_PIPE mode of wireshark-vnc-docker, so CAPTURE_WIRESHARK_DOCKER_IMAGE (default ghcr.io/srl-labs/wireshark-vnc-docker:latest) must be an image that has it; the current :latest does. With an older image no pipe appears, and the session fails within a minute (see below).

Stopping and restarting the capture inside the session's Wireshark works: each time Wireshark opens the pipe, the server starts a new capture into it, and stops it when Wireshark closes the pipe. Closing the session, or its expiry (CAPTURE_WIRESHARK_SESSION_TTL), stops the capture in the pods and the container. When the capture fails (for example when the node's pod is gone, or the lab was destroyed), the server stops the session's container and its ready and vnc routes answer 500 with the reason until the session is closed or expires.