Skip to content

feat: add Prometheus metrics for backup recovery window - #69

Open
ermakov-oleg wants to merge 1 commit into
operasoftware:mainfrom
ermakov-oleg:feat/prometheus-metrics
Open

feat: add Prometheus metrics for backup recovery window#69
ermakov-oleg wants to merge 1 commit into
operasoftware:mainfrom
ermakov-oleg:feat/prometheus-metrics

Conversation

@ermakov-oleg

Copy link
Copy Markdown
Contributor

Summary

Port of upstream #459, #467

Problem: No observability into backup health — operators had no way to alert on stale backups or monitor recovery point objectives (RPO) without manually querying pgBackRest.

Fix: Implements the cnpg-i Metrics service, exposing two Prometheus gauges:

  • cnpg_pgbackrest_first_recoverability_point — unix timestamp of the earliest restore point (first successful backup stop time)
  • cnpg_pgbackrest_last_available_backup_timestamp — unix timestamp of the most recent completed backup (latest backup stop time)

These allow standard Prometheus alerts like "no backup in last 24h" or "RPO exceeds 1h".

Implementation:

  • New MetricsServiceImplementation in internal/cnpgi/instance/metrics.go
  • Registers TYPE_METRICS capability in plugin identity
  • Collect() calls pgbackrest info to get the backup catalog, then delegates to getRecoveryWindow() which uses catalog.FirstRecoverabilityPoint() and catalog.GetLastSuccessfulBackupTime() — these methods filter out errored backups (Start=0 or Stop=0) and use Time.Stop for recoverability
  • Returns 0 for both metrics if no backups exist or credentials fail (graceful degradation)

Unit tests in metrics_test.go cover: nil/empty catalog, single backup, multiple backups, errored backups filtering, all-errored catalog.

Related issues

Signed-off-by: ermakov-oleg <ermakovolegs@gmail.com>
@ermakov-oleg

Copy link
Copy Markdown
Contributor Author

Hi @Agalin, just following up on this PR - would you have a chance to review it when you have time? The changes from all my PRs have been running in our production for a while now without issues, but I’m happy to adjust anything if needed.

@Agalin

Agalin commented May 20, 2026

Copy link
Copy Markdown
Collaborator

Sorry for the late reply, my team was jumping from P1 to another P1 for the last few months. I'll discuss this solution with the team. Our original plan was to try to embed pgbackrest exporter in the plugin to get the same set of metrics it provides - and compatibility with its Grafana dashboards.

@melancholictheory

Copy link
Copy Markdown
Contributor

We would find these metrics useful. We run this plugin for our DBaaS PostgreSQL fleet, and today there is no way to alert on a stale backup without shelling into a pod and running pgbackrest info, which is what this PR fixes.

On the exporter idea, the two do not look like they are in conflict. This PR goes through the cnpg-i Metrics service, so the values land on the instance metrics endpoint CNPG already exposes, with no extra process, port or scrape config. Embedding pgbackrest_exporter means another process in the sidecar with its own endpoint, calling pgbackrest info on its own schedule. It does buy a much wider metric set and the ready made Grafana dashboards, so it is a reasonable longer term goal, but it is also a lot more work than what is already written here.

The names would not collide either: this uses cnpg_pgbackrest_* and the exporter publishes pgbackrest_*, so landing this now does not paint the exporter into a corner.

Did the team discussion reach a decision? Happy to help with testing if that is what is holding it up.

@melancholictheory

Copy link
Copy Markdown
Contributor

One correction to what I wrote above, having looked at the exporter code. Embedding it would not be a separate process: backrest is a normal package, so the plugin can import it and call GetPgBackRestInfo on a timer inside the existing sidecar. The catch is elsewhere. Its metrics are registered with promauto into the default registry and served by its own promhttp.Handler(), so getting them onto the endpoint CNPG already exposes means gathering from the registry and translating into the cnpg-i Metrics response. Without that bridge it is a second port and a second scrape target, in the sidecar or in a container of its own.

If the team does settle on the exporter route, we are happy to take that on and bring the PR. Just say which shape you want, the bridge into cnpg-i Metrics or a separate endpoint, and whether you would rather have it on top of this PR or instead of it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants