feat(analyzer): add Czech PII recognizers (CZ_*)#2155
Open
santiagotri wants to merge 1 commit into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds first-class Czech (cs) analyzer support by introducing new pattern-based CZ_* recognizers (plus Czech DATE_TIME notation support), wiring them into the predefined registry/configuration, and documenting how to run a bilingual EN+CS pipeline.
Changes:
- Added 5 Czech PII recognizers (
CZ_BIRTH_NUMBER,CZ_BANK_ACCOUNT,CZ_ID_CARD,CZ_PASSPORT,CZ_DRIVER_LICENSE) and a CzechDATE_TIMErecognizer (CzDateRecognizer). - Registered the new recognizers in
predefined_recognizers/__init__.pyandconf/default_recognizers.yaml(disabled by default,country_code: cz). - Added documentation and recipes for Czech language support, plus comprehensive unit and end-to-end tests.
Reviewed changes
Copilot reviewed 20 out of 20 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| presidio-analyzer/tests/test_czech_language_support.py | End-to-end Czech pipeline test + PHONE_NUMBER collision regression coverage |
| presidio-analyzer/tests/test_cz_passport_recognizer.py | Unit tests for CzPassportRecognizer patterns/validation |
| presidio-analyzer/tests/test_cz_id_card_recognizer.py | Unit tests for CzIdCardRecognizer patterns/validation |
| presidio-analyzer/tests/test_cz_driver_license_recognizer.py | Unit tests for CzDriverLicenseRecognizer patterns/validation |
| presidio-analyzer/tests/test_cz_date_recognizer.py | Unit tests for CzDateRecognizer Czech date notation coverage |
| presidio-analyzer/tests/test_cz_birth_number_recognizer.py | Unit tests for CzBirthNumberRecognizer (date semantics + mod-11 / exception) |
| presidio-analyzer/tests/test_cz_bank_account_recognizer.py | Unit tests for CzBankAccountRecognizer (weighted mod-11 + invalidations) |
| presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/czechia/cz_passport_recognizer.py | New Czech passport recognizer implementation |
| presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/czechia/cz_id_card_recognizer.py | New Czech ID card recognizer implementation |
| presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/czechia/cz_driver_license_recognizer.py | New Czech driver license recognizer implementation |
| presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/czechia/cz_date_recognizer.py | New Czech date notation recognizer emitting DATE_TIME |
| presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/czechia/cz_birth_number_recognizer.py | New rodné číslo recognizer with checksum/exception handling |
| presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/czechia/cz_bank_account_recognizer.py | New Czech bank account recognizer with validation + date-fragment invalidation |
| presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/czechia/init.py | Czechia country-specific recognizers package marker |
| presidio-analyzer/presidio_analyzer/predefined_recognizers/init.py | Exposed new Cz*Recognizer classes via public predefined recognizers API |
| presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml | Added Czech recognizers to default registry config (disabled by default) |
| docs/supported_entities.md | Documented Czechia entity types and Czech DATE_TIME coverage |
| docs/recipes/czech-language-support/spacy_en_cs.yaml | Bilingual EN+CS spaCy engine config (en_core_web_lg + xx_ent_wiki_sm) |
| docs/recipes/czech-language-support/README.md | Usage recipe and guidance for Czech-language pipelines |
| CHANGELOG.md | Added unreleased entry describing Czech recognizers + recipe |
Comment on lines
+136
to
+153
| def _resolve_conflicts(results: List[RecognizerResult]) -> List[RecognizerResult]: | ||
| """ | ||
| Keep the highest-scoring result on overlapping spans. | ||
|
|
||
| Mirrors the score-based conflict resolution presidio-anonymizer | ||
| applies before operating on the text (higher score wins; on equal | ||
| scores the longer, earlier span wins). | ||
| """ | ||
| ordered = sorted( | ||
| results, key=lambda r: (-r.score, -(r.end - r.start), r.start) | ||
| ) | ||
| kept: List[RecognizerResult] = [] | ||
| for result in ordered: | ||
| if all( | ||
| result.end <= other.start or result.start >= other.end for other in kept | ||
| ): | ||
| kept.append(result) | ||
| return sorted(kept, key=lambda r: r.start) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds 5 new pattern-based recognizers for Czech-language PII plus Czech
DATE_TIMEdate coverage, covering the most relevant entity types under GDPR / zákon č. 110/2019 Sb.,
and fixes the collision where Czech identifiers (rodné číslo, bank accounts) were
mis-detected as
PHONE_NUMBER:[prefix-]number/bank code)CzDateRecognizerAlso ships a Czech language support recipe (
docs/recipes/czech-language-support/)with a ready-to-use bilingual EN+CS NLP configuration (
spacy_en_cs.yaml).Design decisions
PatternRecognizerpattern and are placed inpredefined_recognizers/country_specific/czechia/, mirroring the Germanimplementation from feat(analyzer): add German PII recognizers (DE_*) #1909
enabled: false) following the project conventionfor country-specific recognizers; registered for
supported_languages: [cs]only,with
country_code: czpass promotes to
MAX_SCORE, but a checksum failure on an otherwise date-plausiblenumber returns
None(keep base score, let context words decide) rather thanFalse. Rationale: legitimate rodná čísla exist that fail strict mod 11 — 9-digitpre-1954 numbers carry no check digit at all, and ~1000 numbers issued 1954–1985
fall under the historical mod-10 exception — so a hard reject would drop real PII.
Falseis reserved for structurally impossible values (all-zero suffix/account,bank code
0000, wrong length)PhoneRecognizerscores a flat 0.4 and previouslywon on
850412/0003and on part of19-123456789/0800. Base scores plus contextboosting put the Czech entities at 0.55–0.85 (1.0 on valid checksums), so
score-based conflict resolution (as applied by presidio-anonymizer) picks the
Czech entity; the genuine phone number
+420 601 234 567is unaffected"rodného", …) because the
LemmaContextAwareEnhancersubstring-matches lemmas andno Czech lemmatizer is available in the recommended pipeline
(?<![\w/-])…) instead of bare\bto preventmatches inside bank accounts, IBANs, and longer digit runs
CZ_PASSPORT/CZ_DRIVER_LICENSEenforce an uppercase prefix invalidate_result(): patterns are matched withIGNORECASE(global flags), soordinary two-letter words before a number ("je 123456") would otherwise match
CzDateRecognizeremits the standardDATE_TIMEentity because no available CzechNER model emits DATE labels;
CZ_BANK_ACCOUNTinvalidates date-likedd/yyyyfragments (e.g.
04/2023)uses the official multilingual
xx_ent_wiki_smfor PERSON/LOCATION/ORG anddocuments a HuggingFace transformers model as the higher-quality upgrade path
global_regex_flags: 26(IGNORECASE | MULTILINE | DOTALL)consistent with the rest of Presidio
Test plan
presidio-analyzer/tests/test_cz_*.py) plusan end-to-end scenario (
test_czech_language_support.py)validate_result()/invalidate_result()tests for thechecksum-based recognizers (valid, historical-exception, failing, and
structurally impossible inputs)
address, phone, e-mail, ID card, passport, driving licence, bank account,
IBAN), asserts every span/entity, includes a before-state regression test for
the PHONE_NUMBER collision, and verifies the exact anonymized output — using
precomputed NLP artifacts so CI needs no model download
python -m pytest presidio-analyzer/tests/test_cz*.pymainbaseline (remainingfailures are pre-existing optional-dependency/model gaps, verified via stash
comparison)
ruff checkpasses with no issuesdefault_recognizers.yamlentries follow the existing country-specificpattern; default en-only registry verified unaffected via
RecognizerRegistryProviderload testdocs/supported_entities.mdupdated with Czechia sectiondocs/recipes/czech-language-support/-
CHANGELOG.mdentry added under[unreleased]Checklist
🤖 Generated with Claude Code