EAI-8203 Support Ubuntu 26.04

[pre]

Jira: https://amd.atlassian.net/browse/EAI-8203

ClusterBloom now installs on Ubuntu 26.04 (resolute). A new entry in
supported_ubuntu_versions was not sufficient; three defects stopped a 26.04
GPU node.

Related

• EAI-8208 Give the JVM the memory limit of its container cluster-forge#814

What was broken on 26.04

The GPU driver codename gate accepted only jammy and noble. AMD
publishes amdgpu-install and amdgpu-dkms for resolute from driver
31.30.0, and the amdgpu-dkms package for 26.04 carries the same DKMS module
version, build and package code as the other releases, so the existing tuple
matches without a change. The older 7.x installer train has no 26.04 packages,
so the gate is now derived from the installer train instead of a flat list.

Ubuntu 26.04 has no linux-modules-extra package. Those modules, amdgpu
included, are in linux-modules. The task installed
linux-modules-extra-<kernel> as a DKMS build prerequisite and apt failed with
No package matching 'linux-modules-extra-7.0.0-30-generic'. Bloom asks apt
for the package now and adds it only when a candidate exists. The stale-kernel
recovery path in gpu_kernel_headers_check.yaml had the same assumption: it
skipped every candidate kernel that had no modules-extra package, which on
26.04 is every kernel.

dkms 3.2 writes a note after the module state, for example
installed (Original modules exist). The check for the module of the running
kernel used grep ': installed$', so a correctly built module read as missing
and the run stopped with a misleading message.

Also in this change

AMD GPU Driver 31.40.1 joins the allowlist, because AMD's current
instructions install it.

The lists of supported drivers in the operator messages are built from the
driver matrix now. The same list was written by hand in three places in
gpu_driver_detect.yaml, and this change would have needed all three edited
again.

Test

Four Digital Ocean nodes, Ubuntu 26.04, kernel 7.0.0-30-generic, one AMD GPU
each, ROCm and the driver removed first so every node was fresh:

• First node completed with exit code 0. Driver 31.40.0 built, the node asked
  for the reboot it needs, and RKE2 v1.34.1 started after it.
• Two control planes and one worker joined.
• All four nodes: dkms status reports the module installed for the running
  kernel, modinfo -n amdgpu selects updates/dkms/amdgpu.ko.zst, the active
  module is 6.19.14.31400000, /dev/kfd exists, and AMD-SMI finds the GPU.
• kubectl get nodes shows four Ready nodes on Ubuntu 26.04 LTS.

[pre]

Test result: PASS for the Ubuntu 26.04 support. Two defects were found beside it, one of them fixed here.

Two independent two-node clusters were built from this branch, one per Ubuntu
release. Each cluster has one control plane and one worker, both GPU nodes, a
nip.io domain and a self-signed certificate, then cluster-forge v2.2.2. The
24.04 cluster is the control: it shows that the change does not break the
release that already worked.

Cluster	Ubuntu	Kernel	Nodes Ready	amdgpu driver	Bloom runs	cluster-forge
A (control)	24.04.4	6.8.0-124-generic	2/2	6.19.14.31400000, DKMS installed	2 per node, rc=0	128 pods, all Running or Succeeded
B (new)	26.04	7.0.0-27-generic	2/2	6.19.14.31400000, DKMS installed	2 per node, rc=0	127 pods, all Running or Succeeded

Both clusters run RKE2 v1.34.1+rke2r1. On all four nodes the selected module is
under updates/dkms/, the active version equals the selected one, /dev/kfd
and /dev/dri/renderD* are present, and amd-smi list finds the one GPU.

Both clusters show 53 of 54 ArgoCD applications Synced and Healthy. The 54th is
kaiwo-config on both: it holds the demo volume kaiwo-system/pvc-user-demo,
no pod mounts it, and a WaitForFirstConsumer volume without a consumer keeps
the application Progressing. This is equal on the two releases.

The four changes

1. Node validation. Check if Ubuntu version is supported is ok on both
   26.04 nodes.
2. The codename gate. The 26.04 nodes hold
   amdgpu-install 31.40.0.0.31400000-2370368.26.04 and two resolute lines in
   /etc/apt/sources.list.d/amdgpu.list. The test plan expected the version
   string 31.40.314000-1; that expectation in the plan was wrong, the package
   is from the 31.40 train.
3. The modules-extra difference. linux-modules-extra-6.8.0-124-generic is
   install ok installed on both 24.04 nodes. On 26.04 apt has no candidate for
   that package, and no run log holds No package matching.
4. The dkms state note. dkms 3.2.2 on 26.04 writes
   installed (Original modules exist), and the check for the module of the
   running kernel accepts it. dkms 3.0.11 on 24.04 writes the plain state.

Defect 1, fixed in this branch (2 commits)

The first 24.04 run stopped after the GPU driver install, at
Build reboot-required marker content:

The lookup plugin 'file' failed: Unable to access the file
'/var/run/reboot-required.pkgs': File not found

The task read that file with lookup('file'), which runs on the playbook
controller. Bloom starts the controller in its own mount namespace, where the
host filesystem is under /host, so the read failed although the stat task
before it found the file on the target. A slurp task reads it on the target
now.

The fault is not caused by the 26.04 change and is present in main. It
appears on 24.04 because the DKMS build prerequisites pull
linux-modules-extra, and with it a new kernel image, which raises the OS
reboot-required flag. Ubuntu 26.04 has no modules-extra package, so 26.04 alone
would never have shown it.

The same task then split the package list on the two characters \n, because a
YAML folded scalar does not make a newline out of that sequence. The marker held
one string with embedded newlines. splitlines() gives one entry per package.
The marker now reads:

"packages": ["linux-image-6.8.0-124-generic", "linux-base"]

Defect 2, moved to its own pull request

The local-path provisioner config named the first node, so a join node got no
storage path and every volume for a pod on it stayed Pending. cluster-forge
stopped at the OpenBao volume on both clusters, so this is not specific to an
Ubuntu release. The fix is #307 (EAI-8205), and the test above ran with that fix
applied. Without #307, this branch alone stops at the OpenBao volume on a
multi-node cluster.

Finding: JVM workloads ignore their memory limit on Ubuntu 26.04

This is not a Bloom defect, but it stops the platform on 26.04 and it needs an
owner.

On cluster B, Keycloak was OOMKilled at its 2Gi limit again and again. That
blocked airm-configure, which never created the
airm-rabbitmq-common-vhost-user secret, so airm-agent and airm-api failed
too. The same chart, the same limits and the same image are healthy on 24.04.

The cause is the JVM. With an equal pod manifest on both releases:

	cgroup limit the container sees	JVM max heap
24.04, kernel 6.8	2147483648	512.00M
26.04, kernel 7.0	2147483648	29.50G

-Xlog:os+container=trace on 26.04 gives the reason:

[debug][os,container] controller cpu is not enabled
[debug][os,container] controller memory is not enabled
[debug][os,container] One or more required controllers disabled at kernel level.

JDK 21 reads /proc/cgroups to decide whether it runs in a container. Kernel 7.0
does not list the memory and cpuset controllers in that file any more, so the
JVM decides it is not containerized and sizes the heap from the host memory of
251 GB. It then passes the 2Gi cgroup limit and the kernel kills it.

This applies to every JVM workload on Ubuntu 26.04, not only to Keycloak.
An explicit heap setting is enough: with
JAVA_OPTS_KC_HEAP="-Xms512m -Xmx1g" Keycloak started, airm-configure
completed, the secret appeared, and cluster B reached the same state as cluster
A. That setting belongs in cluster-forge or in a newer JDK, not in Bloom.

Cluster B still carries that manual setting, and auto-sync on its keycloak
application is off, so the cluster does not match what cluster-forge deploys.

Failures

None left in the scope of this pull request. The four machines hold two running
clusters and are ready to be removed.

[pre]

Test report: three clusters installed at the same time, 2026-08-22

Built from this branch at 44893b4 and stamped EAI-8203-ubuntu-2604-44893b4.
The binary was copied to each node, and bloom version was checked against the
stamp on every node before the run.

Three clusters were installed in parallel, not one after the other. Four of
the seven nodes are GPU machines with Ubuntu 26.04.

Cluster	Nodes	OS	Size	bloom phase 1	cluster-forge phase 2
3 GPU nodes	3	Ubuntu 26.04	large	OK, 20 min	OK, 11 min
1 GPU node	1	Ubuntu 26.04	medium	OK, 8 min	OK, 10 min
2 Kaytoo VMs	2	Ubuntu 24.04	medium	OK, 10 min	OK, 59 min

bloom gave rc 0 on every node of every cluster: one node, two nodes and three
nodes, on Ubuntu 26.04 and on Ubuntu 24.04. The three runs did not interfere
with each other.

Ubuntu 26.04 works, and this is what was measured on the node

Ubuntu 26.04 LTS, kernel 7.0.0-27-generic
amdgpu DKMS 6.19.14-2364437.26.04, "installed" against that kernel
amd.com/gpu=1 allocatable, so kubernetes can schedule the GPU
/opt/rocm present
/dev/vdc 5T ext4 on /mnt/disk0

The GPU hosts are 24 cores, 251 GB, one AMD accelerator [1002:75b0], with
vda 720G as root and vdc 5T given to bloom. GPU_STACK_FAMILY was left
empty, so it resolved to instinct.

On the three-node cluster all three nodes came up as control-plane,etcd,
RKE2 v1.34.1+rke2r1, 3 of 3 Ready.

The driver reboot is handled correctly

Each GPU node rebooted once for the DKMS driver. The sequence in the log is:

Wait for bloom to end, or for it to reboot the node   -> UNREACHABLE
Record that the node rebooted during this phase
Wait for the node to come back
Run bloom again after the reboot

So an UNREACHABLE in these logs is the driver reboot and not a fault. Three
reboots were recorded on the three-node cluster, one per node, and every node
came back and finished. The two Ubuntu 24.04 VMs have no GPU and did not reboot.

The two-node local-path fault is fixed

The two-node cluster is the shape that previously stopped at the OpenBao volume:
one control plane and one worker. This build includes 720860b "Give every node a
local-path storage path" from main.

A sample every 60 s from before phase 2 caught it working:

09:18:10  no resources in cf-openbao yet
09:19:14  openbao-0 1/1 Running, age 32 s, on the WORKER node
          data-openbao-0 Bound, 5Gi, storage class "direct"

That is the failing case exactly, a volume wanted by a pod the scheduler placed
on the join node. Read on the running cluster:

nodePathMap: [ { "node": "DEFAULT_PATH_FOR_NON_LISTED_NODES",
                 "paths": ["/mnt/disk0"] } ]

Previously the pod had no nodeName and the PVC stayed Pending on the same
storage class. The four storage classes are unchanged, so they were never the
cause.

End-to-end result

All three clusters reached verdict OK. Every phase ended failed=0.

The three-node GPU cluster reports 3 of 3 nodes Ready, every pod Running or
Succeeded, and 53 of 54 ArgoCD applications Synced and Healthy. The one that is
not is envoy-gateway-config, OutOfSync because of a single resource,
SecurityPolicy ai-gateway-default-deny, with every other resource of that
application Synced. That is drift and not a failure, and it is not in
cluster-bloom.

The two-node VM cluster also finished OK, at 53 of 54, with only kaiwo-config
still Progressing.

Open items, none of them in this branch

• The two-node VM cluster took 59 minutes in phase 2 against 10 and 11 minutes
  for the GPU clusters. It spent that time on three transient conditions, all
  outside cluster-bloom, and all of which cleared without help: the
  envoy-gateway controller sat at 0/1 Ready for about 35 minutes because its
  own :8081/healthz and /readyz refused connections intermittently, while
  the controller itself worked and served xDS to both proxies, and it is now
  1/1 Running; the AIM discovery images failed to pull with
  error reading from server: EOF; and the Kyverno validating webhook was
  briefly unreachable on 127.0.0.1:9345, which sent several aim-disc jobs to
  BackoffLimitExceeded. Worth knowing as a slow path, not as a defect.
• base-update was not run. It reboots a node that is already in service and
  that needs a decision, so it is left for the operator.

Verdict

No defect of this branch was found. Ubuntu 26.04 installs, the amdgpu DKMS
driver builds against the 26.04 kernel, the GPU is schedulable, and the platform
comes up on top on both a one-node and a three-node GPU cluster.