Implement semantic rule validation for DQ rules (#1169)

[Vsatyam013]

Changes

Implements ruleset-level semantic validation for DQ checks (issue #1169).

Adds a ChecksSemanticValidator (module checks_semantic_validator.py) that inspects a ruleset (declarative check metadata) and reports two classes of issue:

• Duplicate rules — rules with the same function, arguments, criticality and filter.
• Conflicting rules — rules targeting the same function and column(s) but with different arguments (e.g. two is_in_range checks on the same column with different thresholds).

Semantic validation is integrated into validate_checks, load_checks and save_checks via a new semantic_validation_mode parameter:

Mode	Behavior
"warn" (default)	Log a warning per issue and continue.
"fail"	Raise a ValueError listing all issues.
None	Skip semantic validation.

Duplicate and conflicting rules are not necessarily invalid (a duplicate is redundant; a conflict may be intentional), so the default mode only warns; use "fail" to enforce a clean ruleset in CI/CD.

Notes on the implementation:

• Rule identity handles both the nested and flat check forms, the top-level and in-block filter, and for_each_column; column-list comparison is order-insensitive.
• Rules using raw Spark SQL expressions (sql_expression) are not deeply inspected — only structured metadata is compared.
• Semantic validation is best-effort: malformed checks never crash the validator (they are skipped/logged and reported by structural validation instead).

Linked issues

Resolves #1169

Tests

• added unit tests
• added integration tests (negative scenarios for validate_checks, load_checks and save_checks)

Documentation and Demos

• added/updated demos
• added/updated docs
• added/updated agent skills

[CLAassistant]

All committers have signed the CLA.

[github-actions]

All commits in PR should be signed ('git commit -S ...'). See https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits

[ghanse]

Hi @Vsatyam013 , the linked issue focuses more on validating the entire ruleset to detect the following issues:

• Detect if a duplicate rule already exists with the same run config
• Detect if a similar rule exists with different arguments (e.g. a narrower threshold of is_in_range)

Can you make some changes to address these more specifically? I think it should be simpler because the check function inputs are mostly structured. We can document any limitations when users execute checks with Spark expressions.

[mwojtyczka]

@Vsatyam013 do you plan to complete the PR, and address review comments? there was no update since 4 weeks.

[Vsatyam013]

Hi @mwojtyczka, sorry for the delay. I've reworked the implementation based on @ghanse's feedback. The SQL keyword validation approach has been removed and replaced with ruleset-level semantic checks, duplicate rule detection and conflicting rule detection (same function and column with different arguments). The validation is now integrated into validate_checks, save_checks, and load_checks with a configurable semantic_validation_mode parameter (warn, fail, or None). CHANGELOG changes have also been reverted.Can you review it?

[mwojtyczka]

Thank you for the PR. I feft some comments. The engine.py does not compile due to identation issues. Please also update the PR description.

[mwojtyczka]

LGTM - implemented feedback and increased code coverage

requested changes implemented