OPENNLP-1895: Add quantized vector storage and search for static embeddings - #1213

Draft

krickert wants to merge 80 commits into

apache:mainfrom

ai-pipestream:OPENNLP-1895-turboquant

Commits on Aug 6, 2026

OPENNLP-1885: Add opennlp-subword: pure-Java SentencePiece inference with exact original-text spans
krickert
committed
6a71703
OPENNLP-1885: Speed up the encode path 2.3x, parity-checked at every step
krickert
committed
fe4abdd
OPENNLP-1885: Move the subword contract into opennlp-api
krickert
committed
db1c371
OPENNLP-1885: Add WordpieceEncoder and fold the unreleased BertTokenizer into it
krickert
committed
f71db4d
OPENNLP-1885: Document the hand-rolled protobuf reader rationale and refactor trigger
krickert
committed
abbf3e6
OPENNLP-1885: Trim commentary and tighten javadoc per review conventions
krickert
committed
5c78b97
OPENNLP-1885: Tighten javadoc to contracts and document helpers and overrides
krickert
committed
dea3031
OPENNLP-1885: Declare serialVersionUID on SentencePieceTokenizer
krickert
committed
1f3883a
OPENNLP-1885: Trim residual commentary per review conventions
krickert
committed
976326a
OPENNLP-1885: Document subword tokenization in the manual
krickert
committed
bf3cacf
OPENNLP-1885: Make the tokenizer graph serializable with computed UIDs, name the format constants, document every helper
krickert
committed
81b1581
OPENNLP-1885: Guard tokenizer deserialization with an allow-listing ObjectInputFilter
krickert
committed
e430109
OPENNLP-1885: Cite the SentencePiece usage example test in the manual
krickert
committed
b843cf7
OPENNLP-1885: Align null contracts and annotations with the review conventions
krickert
committed
f55fbec
OPENNLP-1885: Address review: checked InvalidFormatException for malformed models, tag helpers, javadoc throws
krickert
committed
8ebca7e
OPENNLP-1885: Address review: validation message style, shared test fixtures, thread safety wording
krickert
committed
80029ca
OPENNLP-1885: Deprecate BertTokenizer as a shim over WordpieceEncoder per review
krickert
committed
397c9ec
OPENNLP-1885: Document fixture regeneration and Utf8Text span mapping
krickert
committed
632d996
OPENNLP-1885: Attribute the double-array literature in DoubleArrayTrie
krickert
committed
9670c34
OPENNLP-1885: Expand fixtures README into a validation tutorial and link it from the manual
krickert
committed
d8b0449

Commits on Aug 8, 2026

Commits on Aug 9, 2026

Commits on Aug 10, 2026

Commits on Aug 16, 2026