cluster-agent ClusterRoleBinding subjects not reconciled — RBAC breaks when a Coroot CR is moved to another namespace

[devantler]

Summary

CreateOrUpdateClusterRoleBinding applies the cluster-agent ClusterRoleBinding with a nil mutate function, so controllerutil.CreateOrUpdate only populates .subjects on create and never reconciles them on an already-existing binding. If a Coroot CR is recreated/moved to a different namespace (same name), the cluster-scoped <name>-cluster-agent ClusterRoleBinding keeps pointing its ServiceAccount subject at the old namespace. The cluster-agent SA in the new namespace is then unauthorized, its embedded kube-state-metrics can't list any objects, and the UI shows "It looks like you use Kubernetes, so Coroot requires kube-state-metrics to combine individual containers into applications" with applications ungrouped.

Root cause

clusterAgentClusterRoleBinding correctly derives the subject namespace from the CR — controller/cluster_agent.go:

Subjects: []rbacv1.Subject{
    {
        Kind:      rbacv1.ServiceAccountKind,
        Name:      cr.Name + "-cluster-agent",
        Namespace: cr.Namespace,
    },
},

But the apply helper passes nil as the mutate fn — controller/controller.go:

func (r *CorootReconciler) CreateOrUpdateClusterRoleBinding(ctx context.Context, cr *corootv1.Coroot, b *rbacv1.ClusterRoleBinding) {
	r.CreateOrUpdate(ctx, cr, b, false, true, nil)
}

In the generic helper, when mutateF == nil the closure only (re)sets the controller reference — it never re-assigns b.Subjects:

f := func() error {
	if retain {
		_ = controllerutil.RemoveControllerReference(cr, obj, r.Scheme)
	} else {
		_ = controllerutil.SetControllerReference(cr, obj, r.Scheme)
	}
	if mutateF != nil {
		return mutateF()
	}
	return nil
}
res, err := controllerutil.CreateOrUpdate(ctx, r.Client, obj, f)

Because controllerutil.CreateOrUpdate first GETs the existing object into b (discarding the freshly-built desired Subjects) and the closure leaves Subjects untouched, the diff is empty → OperationResultNone → the stale subject namespace is never corrected.

For contrast, CreateOrUpdateClusterRole does reconcile its content on every pass — which is why stale rules (e.g. #25) are fixed by upgrading, but a stale subject namespace is not:

func (r *CorootReconciler) CreateOrUpdateClusterRole(ctx context.Context, cr *corootv1.Coroot, role *rbacv1.ClusterRole) {
	rules := role.Rules
	r.CreateOrUpdate(ctx, cr, role, false, true, func() error {
		role.Rules = rules
		return nil
	})
}

Reproduction

1. Install Coroot via the operator in namespace A (CR named coroot). The operator creates ClusterRoleBinding coroot-cluster-agent with subject {coroot-cluster-agent, namespace: A}.
2. Relocate Coroot to namespace B — delete the CR in A, create the same-named CR in B (e.g. moving to a dedicated observability namespace). If the operator pod also restarts during the move, its in-memory instances map starts empty, so the last-instance cleanup Delete(... clusterAgentClusterRoleBinding ...) in Reconcile doesn't fire for the old binding either.
3. The ServiceAccount now lives in B, but the cluster-scoped ClusterRoleBinding subject still says namespace: A.

The cluster-agent's embedded KSM informers are then denied cluster-wide:

"Failed to watch" err="failed to list *v1.Node: nodes is forbidden:
User \"system:serviceaccount:B:coroot-cluster-agent\" cannot list resource \"nodes\" ... at the cluster scope"

…repeated for pods, deployments, statefulsets, daemonsets, replicasets, jobs, cronjobs, services, endpoints, namespaces, persistentvolumes(claims), storageclasses and CRDs. The UI then prompts to Install kube-state-metrics and cannot group containers into applications.

Impact

Any namespace relocation of a Coroot CR silently breaks the cluster-agent's RBAC — and therefore kube-state-metrics and application grouping. The ClusterRole rules remain correct; only the binding's subject is stale. It self-heals only if the binding is manually deleted or patched.

Workaround

# patch the subject in place …
kubectl patch clusterrolebinding <name>-cluster-agent --type=json \
  -p='[{"op":"replace","path":"/subjects/0/namespace","value":"<new-namespace>"}]'
# … or delete it and let the operator recreate it on its create path
kubectl delete clusterrolebinding <name>-cluster-agent

Suggested fix

Reconcile the subjects in a mutate fn, mirroring CreateOrUpdateClusterRole:

func (r *CorootReconciler) CreateOrUpdateClusterRoleBinding(ctx context.Context, cr *corootv1.Coroot, b *rbacv1.ClusterRoleBinding) {
	subjects := b.Subjects
	r.CreateOrUpdate(ctx, cr, b, false, true, func() error {
		b.Subjects = subjects
		return nil
	})
}

Note: roleRef is immutable on (Cluster)RoleBindings, so it should not be re-assigned from a differing value (that would make the update fail). Reconciling Subjects only is sufficient and safe here.

Environment

• coroot-operator: 1.9.5 (the relevant code is unchanged on main)
• coroot-cluster-agent: 1.7.0
• Kubernetes: v1.35.x (Talos Linux)