feat: add Prometheus metrics for backup recovery window - #69
Conversation
Signed-off-by: ermakov-oleg <ermakovolegs@gmail.com>
1250c04 to
331f244
Compare
|
Hi @Agalin, just following up on this PR - would you have a chance to review it when you have time? The changes from all my PRs have been running in our production for a while now without issues, but I’m happy to adjust anything if needed. |
|
Sorry for the late reply, my team was jumping from P1 to another P1 for the last few months. I'll discuss this solution with the team. Our original plan was to try to embed pgbackrest exporter in the plugin to get the same set of metrics it provides - and compatibility with its Grafana dashboards. |
|
We would find these metrics useful. We run this plugin for our DBaaS PostgreSQL fleet, and today there is no way to alert on a stale backup without shelling into a pod and running On the exporter idea, the two do not look like they are in conflict. This PR goes through the cnpg-i Metrics service, so the values land on the instance metrics endpoint CNPG already exposes, with no extra process, port or scrape config. Embedding pgbackrest_exporter means another process in the sidecar with its own endpoint, calling The names would not collide either: this uses Did the team discussion reach a decision? Happy to help with testing if that is what is holding it up. |
|
One correction to what I wrote above, having looked at the exporter code. Embedding it would not be a separate process: If the team does settle on the exporter route, we are happy to take that on and bring the PR. Just say which shape you want, the bridge into cnpg-i Metrics or a separate endpoint, and whether you would rather have it on top of this PR or instead of it. |
Summary
Port of upstream #459, #467
Problem: No observability into backup health — operators had no way to alert on stale backups or monitor recovery point objectives (RPO) without manually querying pgBackRest.
Fix: Implements the cnpg-i Metrics service, exposing two Prometheus gauges:
cnpg_pgbackrest_first_recoverability_point— unix timestamp of the earliest restore point (first successful backup stop time)cnpg_pgbackrest_last_available_backup_timestamp— unix timestamp of the most recent completed backup (latest backup stop time)These allow standard Prometheus alerts like "no backup in last 24h" or "RPO exceeds 1h".
Implementation:
MetricsServiceImplementationininternal/cnpgi/instance/metrics.goTYPE_METRICScapability in plugin identityCollect()callspgbackrest infoto get the backup catalog, then delegates togetRecoveryWindow()which usescatalog.FirstRecoverabilityPoint()andcatalog.GetLastSuccessfulBackupTime()— these methods filter out errored backups (Start=0 or Stop=0) and useTime.Stopfor recoverabilityUnit tests in
metrics_test.gocover: nil/empty catalog, single backup, multiple backups, errored backups filtering, all-errored catalog.Related issues
pgbackrest infoor an equivalent #19