Architecture#
concept
Every node boots the same base image. The node's configuration lists which layers to add. All images, base and layers, come out of one build, from one kernel, with one version number.
what runs on a k8or GPU node ┌────────────────────────────────────────────────────────┐ │ pods container images, own userland │ ├────────────────────────────────────────────────────────┤ │ containerd kubelet · CRI in base │ ├────────────────────────────────────────────────────────┤ │ AI-Ops layer Underlay triage engine optional │ │ GPU layer driver · NVML · Fabric Manager │ │ Fabric layer rdma-core · perftest · NIC tools │ ├────────────────────────────────────────────────────────┤ │ k8or base read-only · signed · every node │ ├────────────────────────────────────────────────────────┤ │ k8or kernel 6.12 LTS · shared by all above │ ├────────────────────────────────────────────────────────┤ │ hardware CPU · GPU · RDMA NICs · TPM │ └────────────────────────────────────────────────────────┘
Components#
| Component | Contents | Present on |
|---|---|---|
| k8or kernel | 6.12 LTS; cgroup v2, overlayfs, erofs, dm-verity, hugepages, IOMMU/VFIO, RDMA core, mlx5, eBPF, and little else | every node |
| k8or base | k8or-init (PID 1), device management, networking, time sync, containerd, runc, CNI plugins, kubelet, machine-config API | every node |
| Fabric layer | rdma-core, libibverbs, perftest, NIC firmware and diagnostic tools, RoCE v2 tuning (ECN, DCQCN, PFC) | GPU, storage and data-loading nodes |
| GPU layer | NVIDIA open kernel modules built for the k8or kernel, libcuda, NVML, nvidia-smi, Fabric Manager, container toolkit, DCGM | GPU nodes |
| AI-Ops layer | Underlay triage engine, telemetry collectors, node MCP server, optional small local model | any node, by choice |
The kernel always carries the RDMA, IOMMU, hugepages and VFIO options; they cost very little. The userspace for each capability is what lives in a layer.
Why one pipeline#
Maintaining two independent operating systems, one for CPU nodes and one for GPU nodes, means every kernel patch, CVE fix, containerd bump and kubelet upgrade is done twice, and the two drift apart. One pipeline avoids that:
- One fix, one place. A kernel patch or kubelet upgrade is made once and reaches every flavor.
- No drift between flavors. CPU and GPU nodes are identical except for the layers they add, so a GPU-only failure has only one place to look.
- Drivers always match. GPU modules are compiled against the exact kernel in the same build and signed with the k8or key.
- Room to grow. A new accelerator, such as AMD ROCm or Intel Gaudi, is one more layer, not another operating system.
Next
How a layer is stored, verified and mounted: What a layer is.