Payments 3/7: Failure Taxonomy, Retries, Partial Outcomes & Payment Reconciliation Engine

[yusuftomilola]

Problem / Context

Issue #1571 handles the happy path and clean duplicates. Real payment traffic also includes declines, timeouts, abandoned checkouts, partial refunds, and webhooks that simply never show up. Without a reconciliation layer, these become silent bugs — a booking stuck PENDING forever, or money captured but never confirmed in our own records.

This is issue 3 of 7 in the payment track.

Objective

Define a complete failure taxonomy as first-class data, implement a scheduled reconciliation job that treats provider truth as authoritative and self-heals drift, and give admins a bounded manual-review path for anything that can't be resolved automatically.

Technical approach

• Failure taxonomy as enum values, not free text: DECLINED, EXPIRED (no confirmation within TTL), PROVIDER_ERROR (5xx/timeout talking to the provider), ABANDONED (user never returned), PARTIALLY_REFUNDED, DISPUTED — each with a defined allowed next-state set inside Payments 1/7: Payment Domain Model, Initiation Flow & Idempotent Transaction Lifecycle #1570's transition guard.
• Reconciliation cron: every few minutes, pull Payments in AWAITING_CONFIRMATION past a short threshold and re-verify directly against the provider (reusing Payments 2/7: Real-Time Confirmation Pipeline & the App/Provider State Boundary #1571's fast-path verify call); resolve to CONFIRMED/FAILED from provider truth, independent of whether a webhook ever arrived.
• Escalation tier: payments still unresolved after a longer threshold (e.g. 24h) are flagged MANUAL_REVIEW and surfaced to admins with a reason — never silently retried forever.
• Retry/backoff utility for our own outbound provider calls (verify, refund): exponential backoff with jitter, capped attempts, and a hard distinction between retryable errors (timeout/5xx) and terminal ones (4xx).
• Partial-outcome support: a refund sub-ledger allowing multiple partial refunds against one Payment, tracking "amount captured" vs "amount refunded" rather than a single boolean.
• Admin recovery actions: force-reconcile-now, mark-resolved-manually (reason required, audited), void.

Detailed scope

• Reconciliation cron provider + scheduling.
• Manual-review query/admin endpoints and a minimal admin list view.
• Refund/partial-refund sub-entity and service methods layered on Payments 1/7: Payment Domain Model, Initiation Flow & Idempotent Transaction Lifecycle #1570's guard.
• Shared retry/backoff utility (also used by the on-chain rail issue).
• Metrics: payments resolved by webhook vs. by reconciliation vs. stuck in manual review (feeds the observability issue).

Important edge cases and failure scenarios

• Provider reports "still processing" indefinitely (e.g. a bank-side hold) — must not flip-flop state on every pass; back off polling frequency the longer it stays pending.
• Reconciliation job crashes mid-batch — must be safe to re-run: idempotent, resumable, no duplicate side effects (e.g. a second confirmation email).
• Refund exceeds the remaining captured amount (double-refund race between two admins) — rejected atomically.
• A provider-side outage makes every verify call fail — reconciliation must distinguish "provider unreachable" from "provider says failed," never mass-flagging everything MANUAL_REVIEW after one bad run.
• A failed payment must release its associated booking hold — must coordinate with booking cancellation without leaking a held slot forever.

Dependencies

Depends on #1570 and #1571 (reuses the transition guard and verify-on-return call).

Acceptance criteria

• A simulated "webhook never arrives" scenario is resolved correctly by reconciliation within its polling window (test).
• Double-refund race test rejects the second refund atomically.
• The manual-review queue correctly excludes payments still within a normal provider-outage retry window (no false positives from a single simulated outage).
• Reconciliation is safe to run twice against the same data with no duplicate side effects (idempotency test).

Definition of done

Cron job running in staging with dashboarded metrics; admin manual-review and recovery actions available and role-restricted; failure taxonomy documented in the payments module README; alert wired for manual-review queue depth exceeding a threshold.

[BigJohn-dev]

Allow me fix this ASAP

[michaelsimeon001]

I can deliver this issue in 1 day ETA

[grantfox-oss]

🦊 GrantFox — @michaelsimeon001 has been assigned to this issue as part of the Third Campaign campaign!

Next steps:

1. Open a Pull Request referencing this issue (e.g., Closes #1572)
2. Your PR will be reviewed by the DistinctCodes maintainers

Good luck! Track your progress on GrantFox.