Skip to content

[Feature] add incremental check #452

@loomts

Description

@loomts

Motivation

During database migration, users need real-time confidence in data consistency before switching traffic. The current offline full-check is too slow and heavy. Integrating lightweight verification directly into the CDC pipeline allows users to verify data correctness immediately as events are synced, enabling safer and faster cutovers.

Solution

Implement an Async Incremental Verification mechanism within the Sinker:

  • Workflow: After Sinker writes success, push data to an async check queue.
  • Resource Isolation: Decouple check tasks into a restricted connection pool to prevent CDC starvation.
  • Output: Produce diff logs and consistency metrics in real-time.
  • Toggling: Support enabling/disabling via a configuration flag during runtime.

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions