Skip to content

EAI-8205 Give every node a local-path storage path - #307

Merged
pre merged 1 commit into
mainfrom
EAI-8205-local-path
Aug 22, 2026
Merged

EAI-8205 Give every node a local-path storage path#307
pre merged 1 commit into
mainfrom
EAI-8205-local-path

Conversation

@pre

@pre pre commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Related:

Bloom wrote the local-path provisioner config with a nodePathMap that names
the first node of the cluster:

{"nodePathMap":[{"node":"ubuntu24-1","paths":["/mnt/disk0"]}]}

A node that joins later got no path. When the scheduler put a pod with a volume
on a join node, the provisioner refused it:

failed to provision volume with StorageClass "direct": config doesn't contain
node ubuntu24-2, and no DEFAULT_PATH_FOR_NON_LISTED_NODES available

The volume stayed Pending, and the pod with it. On a two-node cluster this
stopped the cluster-forge deployment at the OpenBao volume, after every Bloom
task passed.

The config manifest already defaults to DEFAULT_PATH_FOR_NON_LISTED_NODES,
which applies the paths to every node. The task that read the first node name
overrode that default, so this change removes the task and the variable. A node
that joins later needs no change now.

Bloom gives each node the same /mnt/diskN paths from its CLUSTER_DISKS, so
one path list fits the whole cluster.

Test

Found and fixed while testing #306 on two two-node clusters, one Ubuntu 24.04
and one Ubuntu 26.04. Both showed the fault, so it is not specific to an Ubuntu
release.

After the change, on both clusters:

  • The config holds
    {"nodePathMap":[{"node":"DEFAULT_PATH_FOR_NON_LISTED_NODES","paths":["/mnt/disk0"]}]}.
  • Bloom's own storage test volume is created on the join node:
    Volume pvc-... has been created on ubuntu24-2:/mnt/disk0/....
  • cluster-forge v2.2.2 deploys to the end, and every volume binds.

Jira: https://amd.atlassian.net/browse/EAI-8205

The local-path provisioner config named the first node of the cluster and
gave the paths only to it. A volume for a pod that the scheduler puts on a
join node then stays Pending, and the provisioner reports "config doesn't
contain node <name>". ClusterForge stops at the OpenBao volume.

The config manifest already defaults to DEFAULT_PATH_FOR_NON_LISTED_NODES,
which applies the paths to every node, and a node that joins later needs no
change. Bloom gives each node the same /mnt/diskN paths from its
CLUSTER_DISKS, so one path list fits the whole cluster.

@lauri-amd lauri-amd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's an assumption that the all the nodes have the same amount of disks available at the same paths?
If the initial has 2 disks the CM is going to be /mnt/disk0, /mnt/disk1 regardless of how many the other nodes have. The other one might have just single disk at /mnt/disk0 .
I think this is still a better than the current state, ✅

@pre
pre merged commit c791258 into main Aug 22, 2026
5 checks passed
@pre
pre deleted the EAI-8205-local-path branch August 22, 2026 05:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants