feat: signed, ordered, exactly-once webhook delivery

[Chucks1093]

Closes #1383

Summary

• Exactly-once delivery — Soroban events get a content-addressed canonical_event_id (SHA-256 of ledger:txHash:eventIndex) stored in a new webhook_events table. Delivery rows use ON CONFLICT DO NOTHING on (subscription_id, canonical_event_id) so retries and concurrent workers never duplicate.
• Ordered delivery — A per-subscription sequence counter (incremented inside SELECT FOR UPDATE) assigns gap-free monotonic subscription_sequence integers. Consumers can enforce in-order processing and detect missed events without any server-side coordination.
• HMAC-SHA256 signed delivery — Every outbound request carries X-Webhook-Id, X-Webhook-Subscription-Sequence, X-Webhook-Timestamp, X-Webhook-Nonce, X-Webhook-Key-Id, and X-Webhook-Signature: v1=<hex>. The signing message is timestamp.nonce.rawBody, protecting against body-swap and replay attacks within a configurable 5-minute window.
• Key rotation — Subscriptions have dedicated signing keys (webhook_signing_keys table). POST /admin/webhooks/:id/keys/rotate transitions the old key to retiring (valid for 24 h to drain in-flight retries) and activates a fresh key immediately.

Files changed

File	Change
backend/migrations/1802000000000_webhook-signed-ordered-delivery.js	New tables: webhook_events, webhook_signing_keys, webhook_subscription_sequences, webhook_nonces; new columns on webhook_deliveries
backend/src/services/webhookSigner.ts	HMAC sign/verify, canonical event ID derivation, key provisioning/rotation
backend/src/services/webhookDispatcher.ts	enqueueEvent (idempotent fan-out), dispatchDelivery (in-flight lock, send, retry/dead-letter), fetchDueDeliveries
backend/src/services/eventIndexer.ts	Calls enqueueEvent after each newly inserted event
backend/src/controllers/indexerController.ts	New: getWebhookDeliveryLedger, rotateWebhookSigningKey, streamWebhookDeliveries (SSE)
backend/src/routes/adminRoutes.ts	Registers /deliveries/ledger, /deliveries/stream, /keys/rotate endpoints
frontend/src/app/[locale]/admin/webhooks/[id]/deliveries/page.tsx	Admin UI: ordered delivery ledger table with live SSE updates + key rotation
docs/webhook-consumer-recipe.md	Consumer guide: signature verification, idempotent+ordered processing, replay guard, rotation, ledger API

API additions

GET  /api/admin/webhooks/:id/deliveries/ledger   # cursor-paginated ordered ledger
GET  /api/admin/webhooks/:id/deliveries/stream   # SSE real-time delivery status
POST /api/admin/webhooks/:id/keys/rotate         # rotate HMAC signing key

Migration notes

The migration is additive: it extends webhook_deliveries with IF NOT EXISTS columns and creates new tables. Existing rows will have NULL for the new columns and continue functioning through the legacy webhookService.dispatch path. New events ingested after this migration will flow through both pipelines.

[ogazboiz]

there's solid engineering in here (content-addressed event ids, ON CONFLICT idempotency, HMAC with key rotation), but it needs a round of changes before it can land:

1. the new migration is numbered 1802000000000, which collides with two existing migrations at that number. renumber to 1803000000000 or higher.
2. header naming: existing outgoing webhooks sign with x-remitlend-signature: sha256= (webhookService.ts:261). this PR introduces a parallel X-Webhook-Signature: v1= plus five more X-Webhook-* headers. align on x-remitlend-* naming, at minimum the signature header.
3. the ordering guarantee is overstated: sequences are assigned at enqueue but dispatch has no per-subscription serialization, so a retried seq N is overtaken by N+1 while the consumer doc says to reject gaps (that would wedge consumers). also ON CONFLICT DO NOTHING after incrementing the counter burns sequence numbers, breaking the gap-free claim. either serialize per subscription or correct the documented contract.
4. webhook_nonces is a new postgres idempotency table nothing ever reads or writes; repo convention is unique constraints plus redis. drop it.
5. zero tests for ~600 lines of new backend service code. add unit tests for the signer (sign/verify/rotation) and dispatcher (idempotent enqueue, retry, dead-letter). remember the ESM pattern: jest.unstable_mockModule before the dynamic import, and mocks must export everything the consumer imports.
6. deliveries locked in inflight are stuck forever if the worker dies (no reaper), and events currently flow through both the legacy webhookService.dispatch path and the new dispatcher, doubling deliveries. gate or migrate the legacy path.
7. fix the lint errors in your own new code (prettier and prefer-const in the new controller/routes) and rebase on current main now that its CI is being fixed.

if you want to keep contributing, join us on Telegram: https://t.me/+DOylgFv1jyJlNzM0