feat: add scalable real-time CTR ROI reduction with tracking and uncertainties

[pecomyint]

Problem

DashPVA already provides fixed-ROI sums, live ROI histories, offline plots against motor metadata, and generic A−B ROI arithmetic. These pieces do not yet form a reproducible CTR reduction: there is no scaled background model, uncertainty propagation, failure-safe peak tracking, metadata-driven correction chain, or corrected per-scan-point export.

Implement one analysis path that produces the same result live and offline and can sustain detector-rate operation without retaining full frames.

Scientific contract

Each result must retain the raw inputs and intermediate values needed to audit the correction. No normalization or geometric correction may be inferred from a PV name or enabled silently.

For a detector-corrected pixel value (y_p) with variance (v_p), signal pixels (S), background pixels (B), and effective unmasked pixel counts (A_S), (A_B), the baseline constant-background estimator is

[
I_\mathrm{net}=\sum_{p\in S}y_p-\alpha\sum_{p\in B}y_p,\qquad
\alpha=A_S/A_B
]

with independent-pixel statistical variance

[
\sigma_\mathrm{net}^2=\sum_{p\in S}v_p+\alpha^2\sum_{p\in B}v_p.
]

Requirements:

• Support signal plus one or more background regions (sidebands or annulus), masks, clipped ROIs, non-finite pixels, and saturated-pixel rejection.
• Phase 1 supports the scaled constant model. A robust constant/planar weighted fit may be added without changing the result schema; fitted-background covariance must contribute to the uncertainty.
• Variance sources are explicit: Poisson counts with gain/read-noise when available, supplied variance, or an “approximate/unknown” quality flag. Never use (\sqrt{I_\mathrm{net}}) for background-subtracted or negative values.
• Normalize by a declared denominator, not a guessed convention. Configuration must distinguish integrated monitor counts from monitor rate × exposure to avoid applying exposure twice:
  [
  I_\mathrm{norm}=I_\mathrm{net}/D.
  ]
• Apply optional calibrated factors through an ordered correction interface:
  [
  I_\mathrm{corr}=I_\mathrm{norm}\prod_j C_j.
  ]
  Polarization, Lorentz/scan-mode, footprint, absorption, solid-angle, and detector-efficiency factors are off by default, named, unit-tested, and recorded per point with their convention and uncertainty. Missing metadata skips the factor and raises a quality flag; it must not fabricate a value.
• Preserve statistical uncertainty separately from supplied systematic uncertainties.
• Keep raw sum, estimated background under the signal ROI, net intensity, normalized intensity, corrected intensity, uncertainties, and all applied factors.

Full CTR structure-factor/model fitting is out of scope.

Peak tracking

Use separate search, integration, and background geometries:

1. Seed the peak manually or from configuration.
2. Estimate the center only inside a bounded search window after background removal (centroid initially; optional local 2-D fit later).
3. Gate updates by minimum SNR, valid-pixel fraction, fit quality, and maximum displacement/velocity.
4. Move the fixed-shape integration/background geometry together. Do not resize the integration ROI based on noisy intensity.
5. On a failed estimate, freeze the last valid position or revert to the configured fixed ROI; never jump to an unrelated bright feature.
6. Record measured/predicted center, displacement, method, confidence, and flags such as peak_lost, roi_clipped, saturated, missing_monitor, and frame_gap.

Tracking must be optional and deterministic so archived frames replay identically.

Result and scan semantics

Emit one compact result per accepted detector frame with:

• scan/run ID, scan-point ID when supplied, frame ID, timestamp, and selected scan coordinates;
• ROI/background geometry and valid/masked areas;
• peak center and tracking diagnostics;
• raw, background, net, normalized, and corrected intensity;
• statistical/systematic uncertainty, units, correction factors, and quality flags;
• configuration snapshot/provenance.

Do not collapse frames merely because motor positions compare equal. If multiple exposures belong to one point, group only by an explicit scan-point identifier and retain the constituent frame results.

Architecture and scalability

• Add a GUI-independent, pure-NumPy CTRReducer in src/dashpva/utils with typed configuration/result objects. The same reducer is used by live and offline paths.
• Wrap it in a stateful single-stream HPC consumer that receives metadata-associated frames and publishes only the compact result record. Detect gaps/out-of-order frames instead of silently losing them.
• Add a BaseDock-owned CTR panel for ROI setup, tracking state, quality indicators, and the live curve; keep the viewer window a thin coordinator.
• Put runtime configuration in settings.py and DB profile defaults/seed scripts. Do not hardcode PVs, thresholds, or correction assumptions.
• Route HDF5 persistence through utils/hdf5_writer.py. Support lossless tabular HDF5 export and CSV export with units, flags, factors, and configuration provenance.
• Precompute ROI masks/coordinates, operate only on bounded ROI/search pixels, avoid full-frame copies, publish O(1) data per frame, use a bounded GUI history, and batch disk writes.
• Reduction processes every frame. Plot decimation may affect display only and must be visible to the user.
• Expose throughput, queue depth, dropped/gap count, and processing latency.

Delivery plan

1. Core reducer: constant scaled background, masks, result schema, uncertainty, normalization, synthetic tests.
2. Tracking and streaming: bounded tracker, quality flags, metadata association, compact PVA result, offline/live parity.
3. UI and persistence: CTR dock, motor-axis curve, HDF5/CSV export, configuration/provenance.
4. Correction providers and validation: calibrated optional factors, archived reference scan, live beam test, documentation.

Acceptance criteria

• Uniform and planar synthetic backgrounds recover a known injected signal within the estimator’s stated tolerance.
• The constant-background variance matches the analytic expression and Monte Carlo Poisson trials.
• Scaling incident flux or exposure leaves normalized intensity invariant when the declared denominator is supplied.
• No correction is applied when its required metadata is missing; the output remains usable and flagged.
• A translating synthetic peak is tracked within 0.5 pixel over the configured motion range; blank/saturated distractor frames do not cause a jump.
• Fixed-ROI mode reproduces the current offline ROI sum for equivalent inputs.
• Live and offline reduction of the same frames/metadata agree numerically.
• Frame gaps, duplicate/out-of-order IDs, clipped ROIs, and excessive masked area are surfaced in result flags.
• Memory remains bounded with scan length, no per-frame full-image copy is introduced, and a benchmark on reference beamline hardware sustains the configured detector rate with stable queue depth.
• Export contains raw through corrected values, uncertainties, coordinates, units, flags, correction provenance, and enough configuration to replay the reduction.
• An archived CTR scan agrees with the beamline’s established reduction within a scientist-approved tolerance, followed by a beam-on validation.