EAI-8232 Stop using PreSync hooks for seaweedfs ExternalSecrets

[silokimmo]

Problem

seaweedfs-config reported Synced / Healthy while Secret/seaweedfs-s3-config did not exist, leaving seaweed-s3 stuck in ContainerCreating indefinitely. ArgoCD showed nothing wrong, so the failure was invisible.

Root cause

Two parts, the second non-obvious:

1. Sync hooks are not tracked resources. Both seaweedfs ExternalSecrets carried argocd.argoproj.io/hook: PreSync. ArgoCD excludes hook resources from an Application's resource tree, so selfHeal can never restore them — once the ExternalSecret was gone, nothing brought it back, and its target Secret went with it via ownerReference GC (creationPolicy: Owner). The app kept reporting Healthy because the missing resources weren't in the tree it evaluates. The diagnostic tell is a blank sync status in .status.resources[].

2. Un-hooking alone causes permanent OutOfSync. Once tracked, the ESO CRD defaults three fields inside every spec.data[].remoteRef: conversionStrategy, decodingStrategy, metadataPolicy. ArgoCD computes OutOfSync by comparing live state against predicted live state, and the predicted state drops fields nested inside list items (lists are replaced wholesale, not merged). Live therefore never equals predicted and the app sits OutOfSync forever while Healthy. Note that top-level defaulted fields such as target.creationPolicy do not diff — only fields inside spec.data[].

Changes

• Remove argocd.argoproj.io/hook: PreSync from seaweedfs-es-s3-config and seaweed-admin-es-secret. Sync-waves -2 / -1 are retained, which preserves the original ordering intent (Secret materialised before the Seaweed CR syncs) without the hook's tracking blind spot.
• Add an ignoreDifferences entry for ExternalSecret on seaweedfs-config covering the three CRD-defaulted remoteRef paths, so the now-tracked resources converge.

Testing

Verified end-to-end on two clean clusters built from scratch with cluster-bloom.

Single-node, CLUSTER_SIZE: medium and 3-node all-control-plane, CLUSTER_SIZE: large. The second is a materially different code path, not just a bigger cluster: Longhorn distributed storage instead of local-path-provisioner, two cilium-operator replicas, and consequently SeaweedFS deployed as 3 masters + 3 volume replicas rather than 1 + 1.

Results on the 3-node cluster — 19 checks, 0 failures:

• seaweedfs-config reached Synced / Healthy unaided.
• Both ExternalSecrets present in the resource tree with a real (non-blank) sync status.
• Both SecretSynced; both target Secrets present with correct ownerReferences; s3.json valid.
• seaweed-s3 Running — the reported symptom does not occur.
• Zero OutOfSync applications across all 54 apps.

Regression test, deleting the ExternalSecret and watching recovery with no intervention:

t+0s    ES=0  Secret=0  app=OutOfSync/Missing   <- GC'd via ownerRef, visibly broken
t+11s   ES=1  Secret=1  app=Synced/Healthy      <- selfHeal restored it

11s on the 3-node cluster, ~105s single-node. Under the old hook the same deletion left the app reporting Synced / Healthy with the Secret permanently absent — that contrast is the fix.

Two apps had not converged at the end of the run, both unrelated and confined to their own namespaces: airm Degraded on secret "airm-rabbitmq-common-vhost-user" not found (no seaweedfs or s3 references anywhere in the airm namespace; also Degraded before this change), and envoy-gateway-config Progressing while still settling.

Follow-up (not in this PR)

The same hook pattern remains on gitea-config (gitea-devuser-secret, gitea-oidc-secret), kaiwo-config (minio-credentials-fetcher), keycloak (keycloak-cnpg-user-credentials) and otel-lgtm-stack (grafana-admin-credentials), all carrying the identical latent defect — keycloak-cnpg-user-credentials was observed live with the tell-tale blank sync status. When those are un-hooked, gitea-config and otel-lgtm-stack will also need the ExternalSecret ignoreDifferences entry added, or they will land in the permanent-OutOfSync-while-Healthy state described above. To be handled in a separate ticket.

Refs EAI-8232.

🤖 Generated with Claude Code