fix(envoy-ai-gateway): sync mutating webhook CA for mainline HTTPS gateway (EAI-8292)

[Q-Dub]

Summary

• Embed clientConfig.caBundle in the envoy-ai-gateway mutating webhook from the same self-signed secret material on every Helm/ArgoCD render, preventing API-server ↔ controller TLS drift that blocks Envoy HTTPS data-plane pod creation.
• Add shared scripts/utils/ai-gateway-webhook-health.sh (probe → sync CA from secret → restart ai-gateway-controller).
• OpenShift install.sh now calls the shared script instead of inline bash.

Fixes the k2 incident where https://aiwbui.wookie.silogen.ai/ failed with connection closed despite healthy app pods — Gateway Programmed=False, Envoy replicas unavailable.

Jira: https://amd.atlassian.net/browse/EAI-8292

Test plan

• helm template envoy-ai-gateway sources/envoy-ai-gateway/v1.0.0 --namespace envoy-ai-gateway-system renders MutatingWebhookConfiguration with non-empty clientConfig.caBundle
• Fresh mainline deploy (bloom → ArgoCD): kubectl get gateway https -n envoy-gateway-system shows Programmed=True
• Webhook probe succeeds: scripts/utils/ai-gateway-webhook-health.sh exits 0 on healthy cluster
• Envoy data-plane pods exist: kubectl get pods -n envoy-gateway-system -l gateway.envoyproxy.io/owning-gateway-name=https
• https://aiwbui.<domain>/ completes TLS (307 redirect to sign-in is OK)
• https://argocd.<domain>/ returns 200
• Simulate drift (patch webhook caBundle to garbage) → run health script → probe passes and Envoy data plane recovers after --heal-envoy-data-plane
• OpenShift install path finds script at ${CLUSTER_FORGE_DIR}/cluster-forge/scripts/utils/ai-gateway-webhook-health.sh after tarball extract

[silokimmo]

Tested this on chalupa-491a (RKE2 v1.34.1+rke2r1, single node, cluster size medium), which had drifted into exactly the EAI-8292 state. The diagnosis and the heal both check out — one gap to flag.

Confirmed the root cause matches

Two self-signed CAs generated 4 minutes apart:

	CN	notBefore
secret self-signed-cert-for-mutating-webhook ca.crt	ai-gateway-controller-ca	10:22:31
webhook clientConfig.caBundle	ai-gateway-controller-ca	10:18:17

ReplicaSet FailedCreate: failed calling webhook
"ai-gateway-controller.envoy-ai-gateway-system.svc.cluster.local":
x509: certificate signed by unknown authority ... "ai-gateway-controller-ca"

failurePolicy: Fail meant no pod could be created in envoy-gateway-system, so both Envoy data planes were at 0/1 and both Gateways were Programmed=False reason=NoResources. Every HTTPS endpoint returned connection refused.

ai-gateway-webhook-health.sh worked

--probe-only correctly detected the failure (exit 1). The full heal patched the caBundle, restarted the controller, restarted the https data plane, re-probed clean, exit 0. https went Programmed=True and all endpoints came back (argocd/gitea 200, kc 302, openbao/seaweed-admin/airmui/aiwbui 307).

Gap: restart_envoy_https_data_plane() only selects the https gateway

ai-gateway was broken by the same webhook, but the selector is gateway.envoyproxy.io/owning-gateway-name=https, so it was never restarted. Its ReplicaSet had already exhausted its backoff, so it did not self-recover. I polled for three minutes after the heal:

t+15s  available=0
t+30s  available=0
...
t+180s available=0

Still 0/1, Programmed=False. An explicit rollout restart with owning-gateway-name=ai-gateway brought it up in ~15s.

The problem is that the script exits 0 in this state. The final probe only tests whether the webhook admits pods — which is true — so a cluster can come back half-healed while the script reports success. Same applies to the gateway-programmed.sh platform gate if it only checks https.

Suggested fix:

restart_envoy_data_plane() {
  kubectl rollout restart deployment \
    -l 'gateway.envoyproxy.io/owning-gateway-name in (https,ai-gateway)' \
    -n "${GATEWAY_NAMESPACE}" >/dev/null
  kubectl rollout status deployment \
    -l 'gateway.envoyproxy.io/owning-gateway-name in (https,ai-gateway)' \
    -n "${GATEWAY_NAMESPACE}" --timeout=180s
}

Or select on label existence (-l gateway.envoyproxy.io/owning-gateway-name) to cover any additional Gateways. If the flag is meant to stay https-only, worth renaming it and documenting that other Gateways need separate handling.

Minor

• The PR description references scripts/utils/ai-gateway-webhook-health.sh (summary + last test-plan checkbox), but the file is added at scripts/ai-gateway-webhook-health.sh. The code is consistent — docs/openshift/install.sh uses the correct path — so it's just stale description text.
• heal_ai_gateway_webhook() always restarts ai-gateway-controller. On chalupa the controller was already serving the cert matching the secret (verified via openssl s_client against :9443), so only the caBundle patch was needed and the restart cost ~180s. Harmless, and correct in the general case where the controller holds a stale keypair — just noting the patch alone was sufficient here.
• The caBundle patch persisted across ArgoCD sync loops, as expected: the current chart doesn't render that field, so there's no desired value to revert to.

Checked off from the test plan against a real drifted cluster:

• scripts/ai-gateway-webhook-health.sh exits 0 on healthy cluster
• kubectl get gateway https -n envoy-gateway-system shows Programmed=True
• Envoy data-plane pods exist
• https://aiwbui.<domain>/ completes TLS (307)
• https://argocd.<domain>/ returns 200
• Drift → heal → data plane recovers (with the ai-gateway caveat above)