The GPU Stack#
The driver lives on the host. CUDA lives in the container. The GPU layer carries the driver stack only; each workload's container image brings its own CUDA toolkit, so teams choose their CUDA version without touching the OS.
| On the host · GPU layer | In the container · workload image |
|---|---|
| Kernel modules: NVIDIA open GPU kernel modules, including peer memory for GPUDirect RDMA | CUDA toolkit: runtime, cuBLAS, cuDNN |
| libcuda and NVML: the driver's user-space interface | NCCL: collectives over NVLink and RoCE v2 |
| nvidia-smi, Fabric Manager: management and the NVLink / NVSwitch fabric | Frameworks: PyTorch, JAX, inference servers |
| Container toolkit: injects the host's driver libraries into GPU pods at start | Its own userland: Ubuntu-based, distroless or NVIDIA base images |
| DCGM: health and telemetry |
How the two halves meet#
host (k8or GPU layer) pod (workload image) nvidia.ko · nvidia-uvm.ko ──────▶ /dev/nvidia* (device nodes) libcuda.so · libnvidia-ml.so ───▶ injected by the container toolkit CUDA runtime · cuDNN · NCCL PyTorch / JAX / vLLM
When a GPU pod starts, the container toolkit on the host mounts the driver's device nodes and libraries into it. The pod's CUDA runtime then talks to the host driver, so the driver version on the node must support the CUDA version in the image.
Why the GPU layer is built with the kernel#
NVIDIA kernel modules have to match the exact kernel they were built against. Built in the same pipeline as the base, they always match. Maintained separately, a kernel update can break GPU nodes on reboot, a common failure on general-purpose distributions. With Secure Boot, the modules are signed with the k8or key in the same build.
Why glibc, not musl
NVIDIA's user-space libraries (libcuda, NVML, Fabric Manager) are built against glibc. A musl-based OS must carry a separate glibc island just for the GPU stack. k8or-os uses glibc throughout, at very little cost in size.