Skip to content

Underlay: the AI-Ops Layer#

concept

GPU nodes fail in ways that take an expert hours to piece together: Xid errors in the kernel log, NCCL timeouts, PFC pause storms, ECC errors, a NIC on the wrong NUMA node. Underlay, the AI-Ops triage engine, collects the evidence on every node and explains what went wrong.

Deterministic first, then interpretation#

1 · collect kernel log, DCGM counters, RDMA and NIC stats, PCIe link state, kubelet events

2 · detect deterministic rules turn raw telemetry into flags and statistics

3 · correlate flags are lined up by time, device and topology

4 · interpret the model reasons over the flags only, never over raw telemetry

5 · report a plain-language finding with the evidence behind it

k8or-node-07 — underlay
mars@k8or-node-07:~$ k8or triage --since 30m
FLAG                         SOURCE        DEVICE     FIRST SEEN
pcie_link_downgrade          pcie          gpu3       14:02:11
xid_79                       kernel log    gpu3       14:02:19
nccl_timeout                 kubelet       pod/train  14:03:40
finding: GPU 3 fell off the bus (Xid 79) eight seconds after its PCIe link
         dropped from x16 to x8. Likely riser or slot. The NCCL timeout is a
         consequence, not the cause. advisory · cordon recommended
mars@k8or-node-07:~$ █

illustrative = target behavior of the triage engine.

What it does#

triage
Failure explanation

Fabric-level diagnosis of stalled training jobs: NCCL, RDMA, PFC/DCQCN, NIC-to-GPU PCIe affinity and gang-scheduling (Volcano / Kueue) symptoms.

interface
Node MCP server

Read-only tools such as get_gpu_health, get_fabric_counters, get_kernel_log and explain_config_drift, so agents query nodes in structured form instead of over SSH.

prediction
Predictive health

Classic machine learning on ECC, thermal and link-flap trends flags a GPU or optic before it takes down a 512-GPU job.

Guardrails#

  • Advisory by default. It explains and recommends. Cordon, drain or GPU reset go through the k8or API with a policy check or human approval.
  • Least privilege. Own cgroup, CPU and memory caps, non-root, read-only access to telemetry.
  • Bounded cost. A small quantized model on CPU, or a call to a central inference service in the cluster. Never GPU time taken from workloads.
  • Removable. Not listed in the node config, not on the node.

What it never does

  • Sit anywhere in the boot path
  • Make scheduling or kernel-tuning decisions at runtime
  • Run as root or write to node configuration
  • Act on a node without policy or approval

Why it runs on the host#

Triage matters most when the node is failing: kubelet down, containerd wedged, the GPU driver hung. A DaemonSet dies with the node's Kubernetes stack; Underlay is started by k8or-init and keeps reading the kernel log and DCGM data. Fleet-wide reasoning and dashboards still run as ordinary cluster workloads. See Placement for the full comparison.