test: fix the flaky persists messages system test

[amaanx86]

This closes #2152

Summary Of Changes

This flake is a teardown race, not a timeout.

AfterEach calls Delete and returns right away. But Delete only marks the object, and the operator's finalizer keeps it in Terminating for a couple of seconds. Ginkgo then re-enters BeforeEach, creates the same name, and the API server says no:

object is being deleted: rabbitmqclusters.rabbitmq.com "persistence-rabbit" already exists

So FlakeAttempts(3) is really FlakeAttempts(1). Attempts 2 and 3 never reach the spec body, no matter what broke attempt 1.

Fix is to make teardown wait:

• AfterEach uses a new deleteRabbitmqClusterAndWait helper that deletes and polls until the object is gone. If it times out it prints the deletion timestamp and remaining finalizers, so if a deletion ever does get stuck we can see why from the CI log instead of waiting for it to happen again.
• The create in BeforeEach is wrapped in Eventually too, since the teardown wait can get cut short by the spec deadline.

SpecTimeout goes 3m to 5m. Ginkgo counts the whole attempt against that deadline including AfterEach, and resets it per attempt, so the extra wait needs to fit. 5m is what the TLS spec in this file already uses.

Additional Context

I opened #2152 and @MirahImage raised #2212 off it. It's still in draft and I don't think either change in it helps, so this replaces it.

SpecTimeout wasn't the problem, on the run above the body got to its last assertion in 115s out of 180s. And this line:

Eventually(createRabbitmqCluster(ctx, rmqClusterClient, cluster)).Should(Succeed())

calls the function once and passes the resulting error to Eventually, so it just polls a value that can't change. The create never gets retried. It needs a closure, which is what I've done here.

@Zerpet mentioned the old cluster might be stuck on something. I couldn't find any sign of that. I timed deletion on kind and a single node cluster goes from Delete to gone in 2s, same UID throughout. It really is just that nothing waits those 2s. The leaked clusters do cause knock on slowness though, the old one keeps running and later specs fight it for memory on the same node, so that may be why this test has felt flaky in a few different ways.

Local Testing

make just-unit-tests and make just-integration-tests pass.

For the system test I deployed the operator to a local kind cluster (v1.35.0, cert-manager v1.15.1) and ran the persistence spec with attempt 1 forced to fail so the retry runs every time.

On main the retry reproduces CI almost exactly, BeforeEach fails 15ms and 9ms later with the same 409. On this branch AfterEach finishes its delete and wait in 2.0s and attempt 2 creates the cluster fine. I also ran the old and new teardown as two contexts in the same suite, old one failed on the 409, new one passed on attempt 2.

One caveat, the specs can't fully pass on macOS. The helpers resolve the endpoint to the kind node IP and Docker Desktop won't route that from the host, so assertHttpReady always times out. That's what I used to fail attempt 1. So I've verified the create/delete/retry behaviour locally but not the message persistence bits, those need CI.

[Zerpet]

Looks good overall. I only have one nit pick.