Skip to content

cmd/release: enforce backport ordering across stable branches - #422

Open
aanm wants to merge 1 commit into
mainfrom
pr/enforce-backport-ordering-invariant
Open

cmd/release: enforce backport ordering across stable branches#422
aanm wants to merge 1 commit into
mainfrom
pr/enforce-backport-ordering-invariant

Conversation

@aanm

@aanm aanm commented Jul 29, 2026

Copy link
Copy Markdown
Member

A fix that lands in an older stable branch while still missing from a newer one is an upgrade regression, since upgrades move from an older to a newer minor and the user loses the fix on the way up. Nothing catches this today: a backporter opens the vX.15 backport PR, a bugfix then merges carrying needs-backport for both X.15 and X.14, and the subsequently-opened vX.14 backport picks it up. It ships in X.14 but not X.15, guarded only by the implicit "please stop merging backport PRs" message before a release.

This adds a check to the step-1 pre-check. Taking two stable branches A and B where B is newer, a violation is a merged upstream PR carrying backport-done for A together with either needs-backport or backport-pending for B. It runs relative to the branch being released and covers both directions, so releasing any branch surfaces both the branches it lags and those it is ahead of. Violations hard-block the release, matching the existing release-blocker check rather than the softer opened-backports prompt, with --force still overriding and the offending PRs printed with their search URLs. Only the --maintained-minors (default 3) most-recent branches are compared, plus the released one, so stale needs-backport labels on EOL branches are not reported as violations nobody intends to fix.

Releasing v1.15.5 with a fix that reached v1.13 and v1.14 but not v1.15 looks like this (PR numbers and titles are illustrative):

🏃 Running step "checking for release blockers"
  ✅ All release blockers merged.
  ✅ All backports merged.
  👀 Checking that every backport in v1.15 is consistent with the other actively-maintained stable branches (v1.13, v1.14)
    ⚠️ Found backport ordering violations:
     • present in older branch v1.13 (backport-done/1.13) but still needs backport to v1.15 (needs-backport/1.15):
      https://github.com/cilium/cilium/issues?q=is:pull-request%20is:merged%20label:backport-done%2F1.13%20label:needs-backport%2F1.15%20repo:cilium%2Fcilium
      https://github.com/cilium/cilium/pull/32871 - bpf: fix CT entry leak on service backend removal
      https://github.com/cilium/cilium/pull/33012 - ipsec: fix XFRM state leak during key rotation
     • present in older branch v1.14 (backport-done/1.14) but still needs backport to v1.15 (needs-backport/1.15):
      https://github.com/cilium/cilium/issues?q=is:pull-request%20is:merged%20label:backport-done%2F1.14%20label:needs-backport%2F1.15%20repo:cilium%2Fcilium
      https://github.com/cilium/cilium/pull/32944 - envoy: bump to 1.29.4 for CVE-2024-32475
😩 Error while running step "checking for release blockers": found backport ordering violations. A fix present in one stable branch must also be present in every newer active stable branch. Please ensure the listed pull requests are backported (or that their backport candidates are frozen consistently) before continuing the release process

When the invariant holds it prints "✅ Backport ordering invariant satisfied across all active stable branches." instead, and with --force the violations are still listed but the release continues.

One caveat: detection relies on the lagging branch carrying an explicit needs-backport or backport-pending label, so a fix never labelled for the newer branch at all is invisible to a label-pair query. An explicit frozen backport-candidate list would still be complementary. The check also inherits the pre-release early return in CheckReleaseBlockers.Run, so it does not run for RCs, consistent with the checks around it.

This PR was prepared with AIL:3.

A fix that lands in an older stable branch while still missing from a
newer one is an upgrade regression: upgrades always move from an older to
a newer minor, so the user loses the fix on the way up. Today nothing
catches this. A backporter can open the vX.15 backport PR, a bugfix then
merges carrying needs-backport for both X.15 and X.14, and the
subsequently-opened X.14 backport picks it up. The fix ships in X.14 but
not X.15, and the only signal is the implicit "please stop merging
backport PRs" message before a release.

Add a check to the pre-check step that enforces the invariant. A
violation is a merged upstream PR carrying backport-done/<A> together
with needs-backport/<B> or backport-pending/<B>, where B is newer than A.
The check runs relative to the branch being released and covers both
directions: the released branch lagging behind an older branch, and the
released branch being ahead of a newer one. That way releasing any branch
surfaces both sides of the invariant rather than only the ones behind it.

Violations hard-block the release, matching the existing release-blocker
check rather than the softer opened-backports prompt, since shipping a
known upgrade regression is not something to confirm past by pressing Y.
--force still overrides it, and the offending PRs are printed with their
search URLs so they can be triaged.

Only the --maintained-minors (default 3) most-recent branches are
compared, plus the released one. Cilium maintains a fixed window of
minors, and EOL branches can still carry stale needs-backport labels
that would otherwise be reported as violations nobody intends to fix.

Note this detects a lagging branch only when it carries an explicit
needs-backport or backport-pending label. A fix never labelled for the
newer branch at all is invisible to a label-pair query and would need an
absence-of-label enumeration to catch.

AIL:3
Signed-off-by: André Martins <andre@cilium.io>
@aanm
aanm force-pushed the pr/enforce-backport-ordering-invariant branch from 7c064c9 to eb92049 Compare July 29, 2026 09:02
@aanm
aanm requested a review from joestringer July 29, 2026 09:07
@aanm

aanm commented Jul 29, 2026

Copy link
Copy Markdown
Member Author

Funny enough I had these changes locally and, when running with the v1.20.0, it flagged some PRs that were merged in 1.18 were still open for the v1.20 😄

@aanm
aanm marked this pull request as ready for review July 29, 2026 09:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant