Skip to content

Host vs Kubernetes#

concept

Two separate questions are often mixed together: where software runs, and how it is delivered. Where it runs decides whether it survives a broken node. How it is delivered decides whether it stays immutable, signed and removable.

Two axes#

Delivered as a package Delivered as a layer
Runs in Kubernetes Helm chart, which becomes a DaemonSet n/a: layers are an OS concept
Runs on the host apt / rpm package, started by init signed image, started by k8or-init

A Helm chart is only a packaging format. Once installed, it creates DaemonSets and Deployments that run under kubelet, so a Helm chart and a DaemonSet share the same fate.

What survives which failure#

Failure DaemonSet / Helm Host package Layer
kubelet or containerd down dies runs runs
GPU driver hung, network partitioned usually dies runs runs
Kernel panic, power off dies dies dies

The k8or answer

Underlay runs on the host, so it survives a broken Kubernetes stack. It ships as a layer rather than a host package because k8or-os is immutable and has no package manager. For the last row, nothing inside any OS can help: the cluster-level side of AI-Ops polls the server's BMC out of band over Redfish or IPMI.

Nodes vs containers#

k8or-os runs the node. Containers bring their own userland: a container image carries files and libraries, not an operating system. What every container on a node shares is the node's kernel, which is why kernel settings such as hugepages, RDMA and cgroup v2 matter directly to workloads.

one node
 ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
 │ pod: CUDA   │ │ pod: API    │ │ pod: DB     │   container images, own userland
 └─────────────┘ └─────────────┘ └─────────────┘
 ───────────────────────────────────────────────
   containerd · kubelet
   k8or-os  base + layers                         what we build
 ───────────────────────────────────────────────
   k8or kernel                                   shared by every container
 ───────────────────────────────────────────────
   hardware  CPU · H100 · NICs

Later: a k8or base container image#

The same pipeline can publish a minimal base image for k8or's own workloads. It is assembled at build time, not mounted at boot:

  • Kept: glibc and CA certificates; a fabric variant adds rdma-core and libibverbs for RDMA workloads.
  • Dropped: kernel, init, containerd, kubelet, kernel modules and boot files.
  • The GPU rule carries over: no driver libraries inside; the container toolkit injects them from the host.