feat(ci): close nightly failure issues when the stage recovers

[noodlemctwoodle]

Summary

Adds a report-recovery job to the nightly E2E validation workflow that closes a stage's tracking issue when that stage goes green again. Documents it in Docs/Pipelines/Deploy-Nightly.md.

Why is this change needed?

The nightly workflow had no recovery path. report-failure opens and refreshes an issue on every stage failure, but nothing ever closed one. A failure that had already been fixed stayed open indefinitely, and the nightly-failure label stopped meaning "currently broken".

Issue #28 is the worked example. deploy-defender-detections failed every night from 4 June because the workflow passed -SubscriptionId to Deploy-DefenderDetections.ps1, which accepts only BasePath, IsGov and WhatIf:

Deploy-DefenderDetections.ps1: A parameter cannot be found that matches parameter name 'SubscriptionId'.

The restructure in #27 (a7b2ab2, 7 July 09:50 BST) rewrote that invocation to -BasePath + -WhatIf. The timing is exact: the 7 July nightly ran at 04:22 and failed, the 8 July ran at 04:03 and passed. The stage has been green every night since, including today. But #28 was still open 16 days later carrying 33 "Re-failed" comments and no signal that it was fixed, which is precisely the confusion this job removes.

Worth noting the underlying deploy bug is already fixed and needs nothing here. Verified that today's run genuinely executed the Defender job rather than skipping it, and that the production deploy at sentinel-deploy.yml splats only BasePath and optional WhatIf, so Monday's deploy carries no latent version of it.

What does this change do?

Adds report-recovery: needs the same five deploy jobs, conditional always(), with its own permissions: { issues: write, contents: read }. For each stage whose result is success it finds the matching open issue, comments with a link to the passing run, and closes it.

always() rather than a success condition because recovery is per stage: one stage can go green in the same run where another fails, and both the close and the open should happen.

Two deliberate restrictions:

• Only success counts as recovery. A skipped stage is not evidence that anything was fixed (deploy-infrastructure skips on every run where the workspace already exists), so it leaves any open issue alone.
• Exact title match on label-filtered issues, rather than reusing the fuzzy in:title search from the failure path. Closing the wrong issue is worse than opening a duplicate, so it must not match a human-filed issue with a similar title.

What does this fix or affect?

Refs #28. Deliberately does not close it by hand: leaving it open means the first run after merge closes it automatically, which proves the job works end to end against the real case.

No behavioural change to any existing job. report-recovery only ever closes issues carrying the nightly-failure label whose title matches a stage name exactly.

Type of change

• feat - new capability
• fix - bug fix
• refactor - restructure without behavioural change
• perf - measurable performance improvement
• docs - documentation only
• test - Pester / schema test changes
• chore - dependency bump, version pin, file rename
• ci - workflow / pipeline change
• tune - analytical-rule threshold / severity / filter change

Files changed (high level)

• .github/workflows/sentinel-deploy-nightly.yml - new report-recovery job; header comment updated to describe the recovery path
• Docs/Pipelines/Deploy-Nightly.md - job added to the stage diagram, plus a section covering both restrictions

Pre-merge checklist

• Pester suite passes locally (./Tools/Invoke-PRValidation.ps1 -RepoPath .)
• Bicep build passes locally if Bicep changed (az bicep build --file Infra/sentinel/main.bicep --stdout > /dev/null)
• dependencies.json regenerated if KQL content changed (./Tools/Build-DependencyManifest.ps1 -Mode Generate)
• Cross-platform parity maintained if pipelines/workflows changed (ADO + GitHub both updated)
• Path-scoped instructions still match the touched content type's schema
• No secrets in committed files (env vars, hardcoded tokens, connection strings)
• Commit messages follow conventional-commit format (type(scope): subject + detailed body)
• Documentation updated if the change affects user-visible behaviour

Parity is ticked as not applicable: Pipelines/ has no nightly equivalent, so this is GitHub-only. Bicep, dependencies.json and Pester are unticked because no Bicep, KQL or test files are touched; the CI gate runs them regardless.

Testing

• Extracted the job's script exactly as the YAML parses it, substituted the needs.*.result expressions, and ran it against a stubbed gh. The fixture contained both issue Nightly E2E deploy validation failed: deploy-defender-detections #28 and a decoy whose title shares its full prefix (... deploy-defender-detections (my notes)). It closed 28, left the decoy alone, and rendered the comment correctly with the backticks literal and the blank line preserved.
• Exercised three run shapes: all-green-with-routine-skip, mixed (one stage recovered while another still fails), and all-red. Correct behaviour in each, exiting 0.
• An earlier version used an associative array; local testing showed declare -A needs bash 4. The runner has bash 5 so it would have worked, but the plain-list form has no version dependency and stays testable anywhere, so it was rewritten.
• Workflow confirmed to parse as valid YAML, with report-recovery present and its if / needs / permissions as intended.

Not smoke tested against a live run, since the job only fires on the schedule from the default branch. The first nightly after merge is the real test, and #28 is the case it should close.

Related

• Refs Nightly E2E deploy validation failed: deploy-defender-detections #28
• Root cause fixed by refactor(repo): restructure into Content/Infra/Deploy/Tools and adopt CalVer #27

[Copilot]

Pull request overview

Adds an automated “recovery” path to the GitHub-only nightly E2E deploy validation workflow so that stage-tracking issues get closed once the relevant stage is green again, keeping the nightly-failure label meaningful.

Changes:

• Add a new report-recovery job that runs with always() and closes matching open nightly-failure issues for stages that succeeded.
• Update the nightly workflow header comments to document the new recovery behaviour.
• Extend the nightly deploy workflow documentation to include report-recovery and its restrictions.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
.github/workflows/sentinel-deploy-nightly.yml	Adds report-recovery job to close stage failure issues after recovery.
Docs/Pipelines/Deploy-Nightly.md	Documents the new recovery job and updates the workflow stage description.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Docs/Pipelines/Deploy-Nightly.md:112

• The stage diagram (and the surrounding wording about a linear chain) implies report-recovery runs after and depends on report-failure (via the ▼ arrow). In the workflow, report-recovery does not need report-failure and can run even when report-failure is skipped, so the diagram should show the two reporting jobs as independent follow-ons after the deploy chain completes.

report-failure              Runs only if any prior job failed.
        ▼
report-recovery             Always runs. Closes the tracking issue for
                            any stage that is green again.

.github/workflows/sentinel-deploy-nightly.yml:410

• report-recovery runs gh issue ... commands without checking out the repo and without specifying the target repository (--repo / GH_REPO). When the job runs outside a git checkout, gh can fail to resolve which repo to query/close issues in, which would make this job fail even on an all-green run.

      - name: Close issues for stages that are now green
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        shell: bash
        run: |