Roadmap#

concept

Seven phases, each with a test that proves it is done. Every phase builds in a virtual machine first; GPU and fabric validation move to real hardware once the layers exist.

Key decisions#

Decision	Choice	Why
Base	Own base, not a fork	Forking Talos is fastest, but the result is Talos renamed. k8or-os studies Talos (machined, its config API, its extension model) as a reference.
Build system	Buildroot now, Yocto later	Buildroot builds toolchain, kernel and minimal userland from source in under an hour. Yocto takes over with several hardware targets, long-term maintenance and SBOMs.
C library	glibc over musl	NVIDIA's user-space stack is glibc-linked; glibc from day one avoids a second C library on GPU nodes.
Init	BusyBox first, then k8or-init	BusyBox gets early phases booting; k8or-init then owns layer verification, mounting, machine config and service supervision.
Images	erofs + dm-verity	Compressed, read-only, fast to mount, verified on every read, for the base and every layer.
Kernel	6.12 LTS, tightly configured	Only what the node needs, with GPU modules built against it in the same pipeline.

Phases#

Toolchain & kernelfoundation

glibc cross-toolchain and the k8or 6.12 LTS kernel configuration, built with Buildroot.

boots in QEMU to a minimal initramfs shell

Minimal userlandbase

PID 1, device handling, networking, time sync, read-only erofs root, A/B partition layout.

boots a read-only root with networking and time sync; A/B slots present

Container layerkubernetes

containerd, runc, CNI plugins and kubelet, with cgroup v2 throughout.

a VM joins a kubeadm cluster and runs a pod

API-driven configno shell

The k8or machine-config API replaces shell and SSH access.

a node is configured and upgraded with no shell or SSH

Fabric & GPU layersacceleration

rdma-core and RoCE v2 tuning; NVIDIA open kernel modules, Fabric Manager, container toolkit and DCGM, as signed layers.

the GPU layer loads on its matching kernel and is refused on a mismatch; nvidia-smi and ibv_devinfo pass inside a pod

Trust & updatessecurity

Unified kernel image, Secure Boot, dm-verity, TPM-measured boot, signed A/B updates.

a signed update applies, and a failed boot rolls back automatically

Hardware validationH100 · RoCE v2

Real GPU servers, real fabric, and the AI-Ops layer under real faults.

nccl-tests all-reduce reaches the expected bus bandwidth, the GPUDirect RDMA path is verified, and Underlay explains injected faults correctly

Beyond phase 6

The k8or base container image, further accelerator layers (AMD ROCm, Intel Gaudi), and a move to Yocto for long-term maintenance.