What a Layer Is#
A layer is a separate, signed, read-only filesystem image that k8or-init verifies and mounts over the base at boot. Its files then appear as part of the OS. It is not a package you install, and it is not a container.
On the node's disk#
/dev/nvme0n1
k8or-gpu-1.0-580.erofs
k8or-aiops-1.0.erofs
Each .erofs file is a compressed, read-only filesystem image, similar in idea to an ISO.
Inside the GPU layer#
usr/
├── lib/modules/6.12.x-k8or/ # nvidia.ko, nvidia-uvm.ko, nvidia-peermem.ko
├── lib/ # libcuda.so, libnvidia-ml.so …
├── bin/ # nvidia-smi, nv-fabricmanager, nvidia-ctk
└── lib/k8or/extension.meta # requires kernel 6.12.x-k8or, base 1.0
How the images combine#
The kernel's overlayfs stacks the read-only images into one merged view of /usr:
Remove a layer image and reboot, and it is gone, with nothing left behind. The same base image boots on every node; only the config decides which layers are added.
Boot sequence#
Firmware verifies the UKI. Secure Boot checks the signed kernel image; the TPM records measurements.
Mount the base. k8or-init mounts k8or-base from the active A/B slot, verified by dm-verity.
Read the machine config. The config declares the node's layers, for example extensions: [fabric, gpu]. A CPU node lists none.
Verify every layer. Signature and extension.meta are checked. A layer built for a different kernel or base is refused, so the node never boots with mismatched drivers.
Merge and load. overlayfs merges the layers; nvidia.ko loads; Fabric Manager and the AI-Ops collectors start.
Start the node. containerd and kubelet start last, so the GPU stack is ready before the node ever reports Ready.
Package vs container vs layer#
| Package (apt / dnf) | Container (driver-container style) | k8or layer | |
|---|---|---|---|
| Changes the OS on disk | yes, file by file | no | no, a separate image file |
| Can drift between nodes | yes | somewhat | no, byte-identical everywhere |
| Removal | uninstall, often leaves debris | stop the pod | delete one file, reboot |
| Kernel match enforced | no | checked at runtime, can fail | at build time and at boot |
| Ready before kubelet | yes | no, races kubelet | yes |
| Verified on read | signed on download only | image signing, optional | dm-verity, every read |
A package is how software is built; a layer is how it is delivered
Inside the k8or pipeline, everything is built as packages from source. What ships to the node is a signed layer, because the node has no package manager at all.
The same idea elsewhere#
- systemd-sysext on Flatcar and Fedora CoreOS.
- Talos system extensions: Talos's NVIDIA support ships exactly this way.
- Android: its system, vendor and product partitions are layers in this sense.
k8or-os uses the same mechanism, driven by its own init, with images built and signed in its own pipeline.