fix: match inflected Polish in the fallback full-text search - #60
Merged
Conversation
Closes #59. The entity_search index uses a non-stemming analyzer, so a question in an oblique case never reached a title in the nominative: "semestrze zimowym" did not find "Semestr zimowy". Polish inflects heavily, so this was a common miss rather than an edge case. build_lucene_query now also emits prefix and edit-distance clauses for tokens of 5 characters or more, at boosts of 0.5 and 0.3. "semestrze" becomes semestr*, "zimowym" becomes zimow*. Short tokens stay exact, since a 3-character prefix matches half the graph. Exact phrase clauses keep their length boost, so a true nominative hit still outranks an inflected one. Measured on a 33-node graph, all five oblique-case questions now recover their node at rank 1, all five unrelated questions still return zero rows, and a question about one of a similar pair ("semestr letni" against "semestr zimowy") still ranks the right one first. Nominative scores 10.3 against 1.4 for the oblique form of the same question, so the boost ordering holds.
…e string Two tests compared build_lucene_query's full output, so the inflection expansion broke them even though the phrase clauses they cared about were unchanged. They now pull out the quoted clauses and assert on those.
The issue suggested tuning fallback_min_score upward, since observed real hits scored around 10 against a floor of 0.5. Measuring after the inflection expansion says the opposite: an oblique-case hit legitimately scores about 1.4 where the nominative form of the same question scores 10. A floor picked from nominative scores would drop exactly the matches the expansion recovers, so it stays at 0.5 and both the config and CLAUDE.md now say why. Also replaces the "no Polish analyzer, inflected forms do not match" note with what the expansion actually does, and keeps the pointer that configuring a Polish analyzer at index creation would be the better fix if one ever ships.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #59.
The
entity_searchindex doesn't stem, so an oblique-case question never reached a nominativetitle.
build_lucene_querynow also emits prefix and edit-distance clauses for tokens of 5characters or more. For "Co się dzieje w semestrze zimowym?":
The prefix trims 2 characters with a floor of 4, since Polish endings are mostly 1-3 chars. Short
tokens stay exact, a 3-char prefix matches half the graph. Both expansion boosts are below 1 so an
exact hit still outranks them.
Measured on a 33-node graph
holds
Also re-ran the #57 abstention checks against Neo4j 5.18 and they still pass.
One thing from the issue notes I'd push back on
The notes suggest tuning
fallback_min_scoreup, since real hits were around 10 against a floorof 0.5. After the expansion that isn't safe any more. A legitimate oblique-case hit scores about
1.4, because inflected matches are low-scoring by construction. A floor picked from nominative
scores would drop exactly the matches this change exists to recover.
So I left it at 0.5 and wrote the reason into
graph_config.yamland CLAUDE.md, so it doesn't getraised later on the strength of the original note.
If a Polish analyzer ever ships in the deployed Neo4j build, configuring it at index creation is
the better fix and this expansion can go. That's noted in CLAUDE.md too.
Notes
instead. Same coverage, just not brittle to the extra clauses.