Feature: support backups taken from a standby (pgBackRest TLS multi-host + --backup-standby)

[melancholictheory]

Summary

CNPG's default backup target is prefer-standby, so by default CNPG dispatches scheduled backups to a replica to keep the datafile scan and object-store upload IO off the primary. With this plugin, a backup that lands on a standby fails at stanza-create with pgBackRest exit 56 (unable to find primary cluster - cannot proceed). On a cluster with two or more instances, the only way to get backups working today is to pin spec.target: primary, which puts the full backup IO back on the primary. That defeats the purpose of prefer-standby.

This is a feature request for backups physically taken from a standby, coordinated with the primary the way pgBackRest supports natively via multi-host plus --backup-standby. It comes with a concrete, SSH-less design and an offer to implement it.

I've read the two earlier issues on this (#79 and #83). Both are closed, and neither landed a fix on main. How this proposal relates to them, and why it isn't a re-file, is below.

Current behavior (main at 025fd12, functionally identical to the latest release v0.6.0)

The plugin configures pgBackRest for exactly one local PostgreSQL instance and never generates a second (primary) host:

• AppendStanzaOptionsFromConfiguration calls appendStanzaOptions(ctx, options, 0, …) with a hardcoded db index 0. The maintainer TODO right above it already raises the same question:

  // TODO: There probably should be more entries for replicas. Is it even doable
// to backup from a replica without SSH access or pgbackrest server running on the other node?

  https://github.com/operasoftware/cnpg-plugin-pgbackrest/blob/v0.6.0/internal/pgbackrest/command/commandbuilder.go#L199-L203

• That single instance is wired to the local unix socket /controller/run/. No pgN-host is emitted for a peer:

  https://github.com/operasoftware/cnpg-plugin-pgbackrest/blob/v0.6.0/internal/pgbackrest/command/commandbuilder.go#L214-L227

• The Backup RPC runs CreatePgbackrestStanza unconditionally, before Take, with no check for whether the pod is a standby or in recovery:

  https://github.com/operasoftware/cnpg-plugin-pgbackrest/blob/v0.6.0/internal/cnpgi/instance/backup.go#L121

• The public API type PgbackrestConfiguration (and the archives.pgbackrest.cnpg.opera.com CRD) has no topology, host, or standby fields. It only carries repositories, wal, data, restore, compression, and stanza:

  https://github.com/operasoftware/cnpg-plugin-pgbackrest/blob/v0.6.0/internal/pgbackrest/api/config.go#L324-L353

• The project's own e2e backup fixtures pin target: primary with // TODO: Implement support for standby backups. in two places:

  https://github.com/operasoftware/cnpg-plugin-pgbackrest/blob/v0.6.0/test/e2e/internal/tests/backup/fixtures.go#L153
  https://github.com/operasoftware/cnpg-plugin-pgbackrest/blob/v0.6.0/test/e2e/internal/tests/backup/fixtures.go#L176

Reproduction: any CNPG cluster with two or more instances plus this plugin. Create a Backup or ScheduledBackup with spec.target: prefer-standby (or leave it at the CNPG default). CNPG places the backup on a replica, the phase goes to failed, and the sidecar shows stanza-create exit 56. Setting spec.target: primary makes the same backup succeed.

Relationship to #79 and #83

• Support running backups on a standby (currently fails at stanza-create with exit 56) #79 ("Support running backups on a standby …") is closed. The reporter noted it had been auto-filed and walked it back.
• Backup RPC always runs stanza-create, so target: prefer-standby / standby backups fail with pgbackrest exit 56 #83 ("Backup RPC always runs stanza-create …") was closed by the reporter, not by a fix. backup.go still calls CreatePgbackrestStanza unconditionally on main. On that thread @Agalin wrote: "while I agree that standby backups should be added, it's not as simple as executing pgbackrest on the standby … backup process must be coordinated across primary and standby node." That is correct, and it is the reason for the design below.

This issue picks up from that comment. It is a concrete design for the coordinated case plus an offer to build it, rather than a re-open of the "just skip stanza-create" idea.

Why the minimal "skip stanza-create on a standby" idea isn't enough

One natural first instinct is to guard CreatePgbackrestStanza with pg_is_in_recovery() and skip it on a standby. That does not produce a working standby backup. pgBackRest's backup command itself requires a primary among its configured pgN hosts. It brackets the backup with pg_backup_start() and pg_backup_stop(), and PostgreSQL refuses those during recovery. With only the local standby configured, backup fails with the same unable to find primary cluster (exit 56) that stanza-create hits. So removing the stanza-create step does not get you there. The plugin has to give pgBackRest a connection to the current primary, which means a multi-host setup. This matches @Agalin's point above.

Proposed design (SSH-less, TLS-server multi-host)

pgBackRest's supported path for a standby backup is a two-host config. It talks to both primary and standby, runs start/stop backup on the primary, and copies data from the standby. A distroless sidecar has no SSH, so this uses pgBackRest's TLS server transport, which is the modern alternative to SSH. Sketch:

1. Emit a second pg-host in the config of the pod running the backup (the standby): pg2-host=<current primary>, pg2-host-type=tls, pg2-host-port, pg2-path, pg2-user, and pg2-host-cert-file / pg2-host-key-file / pg2-host-ca-file. pg1 stays the local instance as today.
2. Run a pgBackRest TLS server in each instance sidecar (pgbackrest server plus tls-server-address / -port / -cert-file / -key-file / -ca-file / -auth), so a peer sidecar can drive pg_backup_start and pg_backup_stop on it. Today the sidecar only runs client commands against the local socket.
3. Pass --backup-standby, either as y or mapped from the CNPG target so prefer-standby becomes --backup-standby=prefer, so pgBackRest offloads the data copy to the standby while coordinating control on the primary.
4. Failover-aware primary discovery. The primary is dynamic. The plugin already reads this from CNPG: configuration.Cluster.Status.CurrentPrimary and Cluster.IsReplica() are used in the WAL path (internal/cnpgi/common/wal.go). The backup config-build path can reuse the same signal to fill pg2-host (the current-primary pod, or the -rw Service) and reconcile it when the primary changes.
5. TLS material for the pgBackRest server. This is a separate cert chain from the plugin's existing CNPG-I gRPC client and server secrets. The plugin would need to provision (or deliberately reuse) certs for the pgBackRest server, with client-cert CNs that tls-server-auth accepts.

This also implies a small API addition, for example an opt-in backupStandby knob on the archive config, since PgbackrestConfiguration has nowhere to express any of this today.

Open questions for maintainers

1. Is a PR along these lines something you'd accept, or do you already have a direction in mind?
2. TLS server lifecycle: a long-running pgbackrest server alongside the existing sidecar process, or one started on demand per backup. Any preference?
3. Certs: provision a dedicated pgBackRest-server chain, or reuse and derive from the existing plugin TLS secrets? Any constraint I should respect?
4. Primary discovery: the -rw Service (k8s handles the failover repointing, though TLS CN verification against a Service name needs some thought), or resolving the current-primary pod from Cluster.Status.CurrentPrimary directly?
5. Anything in the CNPG-I sidecar model that makes a peer-to-peer TLS server between instance pods harder than I've assumed?

Happy to prototype this and open a draft PR to make the discussion concrete. I wanted to align on direction, and on the questions above, before writing much code.

---

References: pgBackRest user guide sections on backup from a standby and TLS server setup; the pgstef writeup linked on #83; pgBackRest #1553 and the Crunchy Data writeup linked on #79.

[Agalin]

Thanks for the detailed description. I'm yet to check the MR. Answering the questions:

1. I don't think this is doable in any other way than running pgbackrest servers and doing true pod-to-pod communication here. No reason not to accept a working PR.
2. I'm not sure how feasible is to start the server ad-hoc but I really doubt it's doable with a long-running one due to the need to keep its configuration up to date with changing Archive object state. Definitely a short-lived, on demand server is better.
3. I'd say reusing certs should be fine here - those components have shared identity (same pod - domain name would match, same service account so service mesh identity would match, etc.).
4. IIRC db certs already include both -ro and -rw service domains in SANs and I'm not even sure if they by default include pod's own name (if cert is shared - and I believe it is then quite likely they don't). That means the service usage is preferred. As long as db certs are reused, we can expect (and simply document that expectation) that db nodes have proper SANs for service connections - those are necessary for proper TLS connection from client pods so it's the scenario we should encounder in most cases anyway.
5. I'm not aware of anything that would complicate your life.

[melancholictheory]

Thanks, that settles most of it. We're happy to go with all three: a short-lived on-demand server, reusing the existing db certificates, and service-based addressing. Before reworking the PR I checked how the certificates and services actually look on a CNPG 1.28 cluster, and one thing doesn't line up.

You remembered the SANs correctly. The <cluster>-server certificate has subject CN=<cluster>-rw, and its SANs cover only service domains: <cluster>-rw, <cluster>-ro, <cluster>-external-rw and <cluster>-external-ro, each in the short, .<ns>, .<ns>.svc and .<ns>.svc.cluster.local forms. No pod name, no pod IP. Its extended key usage is TLS Web Server Authentication.

Certificate reuse also splits more cleanly than I expected. <cluster>-replication is already a client certificate (CN=streaming_replica, TLS Web Client Authentication), so it fits the client side together with tls-server-auth=streaming_replica=*, with <cluster>-ca as the CA.

The problem is addressing. Every name in those SANs belongs to a Service, and the Services only publish 5432:

<cluster>-rw   ClusterIP   5432/TCP
<cluster>-ro   ClusterIP   5432/TCP

They're ordinary ClusterIP services, not headless, so connecting to <cluster>-rw:<pgbackrest-port> never reaches a pgbackrest server in the primary's sidecar. The pod itself takes traffic on any port via its IP, but that IP and the pod name aren't in the certificate. So no single name is both covered by the db certificate and able to carry traffic to a non-Postgres port on the primary.

Three ways out, as far as I can see:

1. Issue a separate certificate for the pgBackRest channel, with pod-level SANs or SANs for a headless Service the plugin creates. Addressing becomes simple, but it gives up the certificate reuse you preferred.
2. Keep reusing the db certificates and add a headless Service whose name is in the SAN list. That means changing the SAN list, which is CNPG's territory rather than this plugin's.
3. Keep reusing the db certificates and connect by pod IP, accepting that the certificate won't match the address. Whether that's workable depends on pgBackRest being able to skip host name verification, which I haven't confirmed.

Do you have a preference, or is there an approach I'm missing? Once this is settled I'll rework the PR onto the on-demand server and certificate reuse.

[Agalin]

Hmm. You're correct, CNPG currently doesn't provide any way to extend autogenerated services, only disable them. To be honest, that's something they should address - either add an option to extend via the Cluster CRD or at least via CNPG-i.

We can drop the third option from the get go. That's absolutely the worst option and hardly acceptable to anyone.

So let's talk about what CNPG and CNPG-i actually let us do:

• Cluster CRD supports extending the SAN list.
• It also includes an option to define managed services - and those managed services support the same split of pods as the default ones + should be configurable as headless.
• CNPG-i API includes a mutate cluster hook which allows plugins to emit patches to the cluster resource.

Those options together would let us:

• automatically inject a service that would be handled by the operator itself (ensuring all the labels are always correct),
• automatically inject a new SAN to all pods (using the same cert and CA, all automated by the operator).

That would quite cleanly implement option 2. Is that a good approach? To be honest, I'm not sure, mutating the cluster object is less than ideal. Still, I believe it should be a viable and mostly stable solution - as long as there is an opt-out from each automatic change (or at least a shared opt-out).

That still seems easier than providing a dedicataed CA and duplicating operator's work to ensure service is pointing to a correct node.

The ultimate hack here would be to disable the default rw serrvice (which is possible) and create a replacement with a slightly altered implementation, but I really don't think this would be a good idea. Instead, this limitation should be raised as an issue in the CNPG repository.

[melancholictheory]

Sounds good, we'll go with option 2 via the mutate cluster hook. Options 1 and 3 are out for the reasons you gave.

I checked the certificate and service layout on a live cluster first, and it holds up:

• Managed services cover the headless case. spec.managed.services takes a serviceTemplate, so clusterIP: None gives a headless service with the rw/ro/r selector split, which is what we need to reach a specific primary on a non-Postgres port.
• The db certificates already give us both sides. <cluster>-replication is a client cert (CN=streaming_replica, client auth) that pairs with tls-server-auth=streaming_replica=*, <cluster>-server is the server side with the service-domain SANs you described, and <cluster>-ca is the CA.
• Neither the SAN change nor the managed service restarts anything. Adding a name to serverAltDNSNames isn't in the documented rolling-update triggers, and I confirmed it on a running cluster: the operator regenerated the server certificate online, with no pod restart. So turning the feature on doesn't bounce the instances.

So the shape on our side: the mutate-cluster hook adds a headless managed service on the pgBackRest port plus the matching SAN, each behind an opt-out, reuses the db certs, and starts a short-lived server per backup. I'll rework the PR onto that.

On the underlying limitation (autogenerated services can only be disabled, not extended): I can open an issue on the CNPG repo for it, unless you'd rather raise it yourself since you know where it should land.

[melancholictheory]

Quick follow-up: we opened the CNPG issue for the underlying limitation at cloudnative-pg/cloudnative-pg#11255. It describes the options from this thread (letting the generated services be extended, or covering managed-service names in the server certificate) and links back here.

[Agalin]

Thanks for raising the issue to upstream.

[ardentperf]

Instead of trying to set up custom services or manage custom certs, could we just manually have our code verify that the SAN is what’s expected? I think we know what it should be (-rw)?

[melancholictheory]

Good instinct on the SAN, @ardentperf, that is exactly what makes reusing the existing certs viable. I went and checked pgBackRest to be sure, and the catch is where the verification happens.

pgBackRest does the check itself, inside its own TLS handshake, not somewhere the plugin can hook. In tlsClientAuth() it calls tlsClientHostVerify(host, cert), which matches the certificate SAN (falling back to CN) against the host it dials, i.e. the pg1-host / repo1-host value. So there is no spot for us to "verify the SAN is <name>-rw" in our own code. pgBackRest already verifies it, against whatever address we hand it to connect to.

We also cannot turn the check off to sidestep it. In protocolRemoteExec() (src/protocol/helper.c) the verifyPeer argument to tlsClientNewP() is hardcoded to true, and there is no config option for host-to-host TLS to disable it. The only *-verify-tls toggle is repo-storage-verify-tls, which covers the object store, not pg/repo host connections.

That leaves us needing the certificate to be valid for the exact name we connect on. <cluster>-rw is already in the server cert SAN, but it resolves to the Postgres port (5432), not the pgBackRest sidecar port, so we cannot reuse it as-is. That is why the managed-service route still needs both parts: a headless managed service on the pgBackRest port, plus that service name added to the server cert via serverAltDNSNames, reusing the existing CA and certs.

So I think we still land on option 2. Happy to be wrong if there is a pgBackRest angle I missed here.

[ardentperf]

I see - sounds like there’s no equivalent of sslmode = require (or prefer) 😢

[ardentperf]

Just to clarify - would option 1 with pod-level SAN be possible today with no CNPG changes needed? If so, and if the CNPG changes take a long time to get consensus on, that might be worth keeping in mind as a backup plan?

[melancholictheory]

Right, there's no client-side "require but don't verify" knob like libpq's sslmode=require. pgBackRest either verifies the host against the cert, or (with verifyPeer off, which isn't exposed) checks nothing.

On option 1: yes, it's doable today with no CNPG changes. If the plugin runs its own small CA, it can mint a short-lived server cert per backup with the target pod's address in the SAN (the pod IP is known at backup time via the API, or a pod DNS name) and point pgBackRest at that address. Verification passes because we control the SAN, and CNPG's certs and services stay untouched. So it is a fair backup plan to keep in the pocket.

Two things to weigh before treating it as the fallback:

• It means the plugin owns a CA and the cert lifecycle (mint, rotate, mount into the sidecars), which is the "duplicating the operator's work" cost @Agalin flagged. Pod IPs also move, so the cert has to be minted per backup rather than once.
• Option 2 does not actually wait on a CNPG change. It uses spec.managed.services (a new headless service on the pgBackRest port) plus spec.certificates.serverAltDNSNames (the SAN), both already in current CNPG. I confirmed on a live cluster that adding the SAN regenerates the server cert online with no restart. The CNPG issue (#11255, extending generated services) is only a cleaner long-term shape, not a blocker for shipping option 2.

So my plan is still option 2 as the primary, with the self-signed / pod-SAN route noted as a fallback if the managed-service approach hits something unexpected.