Throttle sdp-expectation-notifications hook and surface backstop notification controls

[vmariiechko]

Related Issue

Closes #(none: notification-hygiene follow-up to the v1.11.0 asset, driven by live investigation)

Summary

The shipped sdp-expectation-notifications hook notifies on every qualifying flow_progress event, and the platform emits expectation counts once per microbatch, so one failing expectation floods on multi-batch, full-refresh, and continuous pipelines (measured live: 4 notifications for one expectation in one update; a fresh event within ~70s of every arriving file in continuous mode). This PR revises the asset around a measured, two-lane design: a time-aware throttled hook (fast lane) and the backstop alert promoted to the guaranteed lane with its platform-native notification controls surfaced and documented.

Every design decision traces to live evidence from 2026-07-17/18 (serverless SDP, non-development-mode target, Free Edition playground); the findings dossier lives outside the repo and the README carries the numbers.

Changes

• Hook: two-layer time-aware throttle, at most one notification per (pipeline, dataset, expectation) per dq_notify.throttle_seconds (default 3600; 0 restores old behavior). In-memory layer always on (timestamp-based, not a seen-set: continuous pipelines keep one process alive for days); optional durable marker files under <dq_notify.state_dir>/dq_notify_state/<pipeline>/<dataset>__<expectation>.json in an existing UC Volume (one human-readable JSON per expectation, overwritten in place). Per-pipeline scoping comes from the event's own origin, zero config, no cross-pipeline suppression. Fail-open on any state error.
• Webhook channels: dq_notify.channel_format = slack (validated E2E: secret scope -> dbutils.secrets.get -> hook -> channel message, HTTP 200) | teams (documented Adaptive Card envelope, marked doc-confirmed, not live-tested) | generic. Secret-scope URL resolution with graceful degradation on a missing scope (validated); {{secrets/...}} config interpolation documented as non-functional in SDP pipeline configuration (observed).
• New prompt state_volume_path (placeholder default = in-memory throttle only). Deliberately no volume resource: operational state should outlive any one bundle.
• Backstop alert: notify_on_ok: true new default (measured: exactly one email per state transition and one recovery email; the recovery email closes the masking window). retrigger_seconds and destination_id ship commented with measured semantics. Measured ERROR behavior documented (emails every evaluation until fixed).
• README/skill/references rewritten around the lived numbers, including corrections: hook_progress records ENABLED/FAILED/DISABLED; teardown grace budget is seconds (20s/event hook lost 6 of 6 queued notifications under natural teardown); INSERT INTO from a hook fails with UNSUPPORTED_SPARK_SQL_COMMAND; volume file I/O works except append; a no-new-data update emits no expectation events.
• Tests: throttle logic exercised offline (exec of the pure helpers against a temp state dir: window suppression, cross-process durability, per-pipeline scoping, marker overwrite, 0-disables, empty-state-dir purity), payload formats and guarded secret resolution asserted, spark.sql AST guard retained.

Change Area

• Asset Library (assets/<name>/)

Configuration Axes Affected

• Asset Library (new asset, asset schema, or framework changes)

Testing

• All tests pass (pytest tests/ -V): 2446 passed, 163 skipped
• New tests added for new functionality

Asset Changes

• Asset installs standalone (validated from this working tree into a fresh wrapper bundle)
• Asset is self-contained (no references to library/helpers.tmpl or other assets)
• tests/configs/assets/sdp_expectation_notifications.json updated with the new prompt
• Asset appears in ASSETS.md catalog (entry updated)

Live E2E validation (2026-07-18, playground, from this branch's working tree)

• Install -> validate -> deploy -> run: first update produced exactly 1 notification (marker: failed 3003 / passed 18929, matching the event log's single expectation event exactly) and 1 Slack channel message via the secret-scope path (HTTP 200).
• Full-refresh rerun inside the window: 0 new notifications, marker untouched (suppression proof with shipped code).
• Backstop alert deployed with notify_on_ok true, evaluated on cron, reached TRIGGERED, delivered email.
• Multi-pipeline layout confirmed: the demo's state folder sits beside the investigation pipelines' folders under one shared volume root.

Not merged pending maintainer review.