OPENNLP-1895: Add quantized vector storage and search for static embeddings - #1213
Draft
krickert wants to merge 80 commits into
Draft
OPENNLP-1895: Add quantized vector storage and search for static embeddings#1213krickert wants to merge 80 commits into
krickert wants to merge 80 commits into
Conversation
…with exact original-text spans New opennlp-extensions module implementing SentencePiece model inference without native code: the ModelProto reader, the model-embedded normalizer (precompiled character map over a Darts-clone double-array trie, whitespace collapsing and escaping, the dummy word-boundary marker), unigram best-path segmentation, BPE agenda merging, byte fallback, and user-defined symbol handling. The public contract is SubwordTokenizer/SubwordPiece; every piece reports the exact UTF-16 span of the caller's original text it came from, and the model normalizer is also exposed as an OffsetAwareNormalizer producing AlignedText. Parity with the reference implementation is asserted, not assumed: five tiny bundled models (unigram, unigram with byte fallback, BPE, identity normalization, whitespace-as-suffix) carry fixtures generated by the sentencepiece Python package over 40 inputs each, checked piece for piece, id for id, span for span, plus each model's embedded self-test samples. An opt-in test (-Dopennlp.subword.eval.dir) runs the same assertions against real downloaded models; T5-small and ALBERT-base-v2 pass exactly, including mixed scripts, emoji ZWJ sequences, BOM, and CRLF inputs.
…step The vocabulary trie dispatches wide nodes (the root and first level of a real vocabulary) through a 256-entry direct table, one load per byte, and scans narrow nodes' short label slices linearly instead of binary searching; a randomized differential test holds both layouts against a map-backed reference, and moving the duplicate-piece detection into the counting pass fixes the index error it previously produced. Non-unknown segments reuse the vocabulary's piece string instead of decoding their bytes, since the trie match means the bytes are identical. The normalizer precomputes, per possible first byte, whether any character-map rule or user-defined symbol starts with it; a clear bit proves the prefix machinery would pass the byte through raw, so plain ASCII text skips it entirely. The per-chunk record became a per-call scratch, the input view keeps its oversized buffers with an explicit length instead of trimming (pure-ASCII text gets an identity offset map and no map array at all), the Viterbi scratch is one interleaved array with scores as raw float bits, and the character-map trie walk relies on the JVM's own bounds checks with the fail-loud translation on the cold path. All 37 bundled parity tests and the T5-small and ALBERT real-model fixtures pass byte-identically. Single-thread throughput on the T5-small vocabulary goes from 2.83M to 6.47M pieces per second, from 0.62x to 1.42x of the reference implementation measured through its Python binding.
SubwordTokenizer and SubwordPiece move to opennlp.tools.tokenize, next to Tokenizer and WordpieceTokenizer, matching where every other seam of this round lives. The opennlp-subword module keeps only the SentencePiece implementation.
…zer into it WordpieceEncoder in opennlp-api runs the full BERT tokenization pipeline as a SubwordTokenizer: every piece carries its vocabulary id and the span of the original text, surviving the normalization steps that change, insert, and remove characters. Content is computed with the same library calls the previous pipeline made; offsets come from a per-code-point rerun, with contextual case mappings (Greek final sigma) falling back to word-wide spans that widen but never misplace. List and map constructors cover line-number and explicit-id vocabularies. BertTokenizer, unreleased and superseded, is removed. The dl tokenizer creation builds on the encoder behind the existing Tokenizer plumbing via a package-private adapter with unchanged special-token selection, and a vocabulary missing its special tokens now fails at construction instead of at the first id mapping, pinned by a test. Parity is enforced twice: a differential suite against the reference pipeline (kept test-only as ReferenceBertPipeline) over a curated corpus plus 800 randomized inputs, and the removed class's reference token sequences ported case for case. WordpieceTokenizer is untouched.
…verrides Applies the review conventions from the OPENNLP-1869 review: class javadoc states the contracts instead of design narrative, every override carries inheritDoc with its null contract, and the private helpers are documented.
The tokenizer is Serializable through the OffsetAwareNormalizer contract but declared no serialVersionUID, which the compiler warns about. Added the serialver-computed value so it matches the convention used across the normalizer classes.
Adds a Subword Tokenization section to the Tokenizer chapter: the SubwordTokenizer contract and its original-text span guarantee, loading and using a SentencePiece model including the OffsetAwareNormalizer face, and the WordpieceEncoder pipeline with its vocab.txt construction and special-token framing.
…s, name the format constants, document every helper
…bjectInputFilter SentencePieceTokenizer gains serialize(OutputStream) and deserialize(InputStream) methods. Reads are filtered through an ObjectInputFilter that allow-lists only the classes reachable from a legitimate tokenizer graph and bounds graph depth, references, and array length; foreign payloads are rejected with InvalidClassException before being materialised. Limits are adjustable through a DeserializationLimits record for unusually large vocabularies; the allow-list is not configurable. The serialVersionUID is recomputed for the new public methods.
Add SentencePieceUsageExampleTest asserting the load-and-encode workflow and point the tokenizer manual section at it.
…ormed models, tag helpers, javadoc throws
…ixtures, thread safety wording - Normalize the argument validation messages to the project style, naming the offending parameter and dropping the leading article and the trailing period, in WordpieceEncoder, SentencePieceTokenizer, ModelProtoReader, BpeEncoder, UnigramEncoder and the SubwordPiece compact constructor. - Stop promising thread safety in the SubwordTokenizer contract and state that it is implementation specific instead; the manual now records that both shipped implementations are immutable and therefore safe for concurrent use. - Drop the @throws IllegalArgumentException tags that only repeated the inherited contract on the normalize and normalizeAligned overrides, leaving a plain {@inheritdoc} as the rest of the class does. - Remove commentary about release history rather than about the code: the pointer to the BertTokenizer class of the 3.0.0 milestone builds in WordpieceTokenizer, and the "frozen" qualifier on the ReferenceBertPipeline baseline. - Move the bundled model loading and the fixture file reading out of SentencePieceParityTest into SentencePieceFixtures, so the alignment, validation and serialization tests no longer reach into another test class for a tokenizer. - Extract MODEL_SUFFIX and FIXTURES_SUFFIX constants on SentencePieceFixtures and use them in SentencePieceRealModelEvalTest when deriving a fixture path from a model path, instead of repeating the two literals. - Fold the five duplicated @valuesource model lists into a single SentencePieceFixtures#models @MethodSource, so adding a bundled model stays a one line change. - Correct the parity test javadoc, which credited a nonexistent gen_fixtures.tsv sibling script instead of the gen_fixtures.py script in the test resources. - Pin accessors that had no coverage: every score is finite and out of range ids are rejected, byte pieces occur only in byte fallback models and always render in the <0x..> form, and isByte rejects negative ids. - Assert SubwordPiece.span() next to start and end in WordpieceEncoderTest so the derived span stays covered by the piece assertions. - Document the IOException of the serialized helper in the serialization test and fully qualify the OutputStream javadoc link now that the import is gone.
… per review Applies the review: the old entry point stays through one stable release instead of being removed, and the DL extension point keeps its descriptor. - Recreate BertTokenizer in opennlp-api as a thin shim, deprecated since 3.0.0 forRemoval, with the original three Set based constructors and the original tokenizePos message. tokenize delegates to encodeToPieces; ids are synthesized from the set order because the tokenize path never reads them. Null contract follows this branch's reviewed convention, IllegalArgumentException, documented in the throws clauses. - Delete the package-private EncoderTokenizer; the adapter now lives in opennlp-api where downstream code can reach it. AbstractDL's protected createTokenizer returns BertTokenizer again, restoring the override descriptor so an already compiled subclass keeps overriding at runtime, and createPipelineTokenizer hands back the shim. - Delete ReferenceBertPipeline and point the curated and randomized differential tests in WordpieceEncoderTest at the shim, pinning shim and encoder to one sequence. Add BertTokenizerTest covering each constructor's argument validation, the default special token chain, and the exact tokenizePos message. Independent expected sequences continue to live in WordpieceEncoderReferenceSequencesTest. - Fix a real divergence the compatibility check surfaced: the encoder kept U+2028 and U+2029 inside words while the old pipeline split on them, so a word carrying a line or paragraph separator became the unknown piece. cleanAndIsolateCjk now maps Zl and Zp to a space, with a span asserting regression test. - Manual: the WordPiece section describes the deprecation and the migration, including the Set to List vocabulary change.
Add a README for regenerating the bundled SentencePiece parity fixtures and clarify that Utf8Text is the encode-path span bridge behind SubwordPiece offsets.
State that the reader is an independent re-implementation of the serialized format, cite Aoe (1989), Yata et al. (2007), and Kanda et al. (2023) in the class javadoc, and decode the bit-9 offset extension with a plain conditional instead of the branchless form.
…ink it from the manual State that the reference implementation produces the expected fixture outputs, add the end-to-end validation steps for the bundled and real models, and point the manual's SentencePiece section at the README.
The absolute GitHub URL 404s until merge and pins the branch layout.
New extension module, targeting a modern (2025) static-embedding distillation format as OpenNLP's word2vec/GloVe successor: same flat per-token vector table artifact shape, pure JVM lookup at inference time, no PyTorch/ONNX runtime dependency. SafetensorsFile/SafetensorsHeaderParser read the safetensors format (8-byte little-endian header length, JSON header describing each tensor's dtype/shape/byte range, then raw tensor bytes). Hand-rolled cursor parser scoped to the header's actual shape, no third-party JSON dependency, matching the project's existing data-file reader discipline. safetensors carries no executable content (unlike PyTorch's pickle-based checkpoints), so no XXE-style hardening is needed, only ordinary malformed-input handling. singleMatrixTensorName() deliberately does not guess a tensor key name convention: distillation tools do not agree on one, so it auto-detects the lone 2-D F32 tensor and fails loud listing every candidate when that is ambiguous, rather than risk silently loading the wrong tensor. Next: tokenizer wiring and the mean-pool/normalize lookup path, targeting minishlab/potion-base-8M as the v1 reference model.
WordPieceVocabulary reads a BERT-style vocab.txt (line number is the token's row id, the format minishlab/potion-base-8M and the wider BGE/BERT family ship). StaticEmbeddingModel wires it to the existing BertTokenizer/WordpieceTokenizer (reused as-is, no new tokenizer code) and the safetensors reader from the previous commit, and implements the pooling formula. The formula is verified against MinishLab's Rust reference implementation (model2vec-rs), not assumed: [CLS]/[SEP] are stripped before pooling since this is table lookup, not transformer input (the tokenizer always adds them, so this class trims the first/last token rather than needing a second tokenizer mode); unknown tokens are dropped from both the sum and the denominator; each pooled token's vector is multiplied by an optional per-token weight from a second "weights" tensor when the safetensors file has one; the sum is divided by the plain pooled-token count, not the sum of weights, which is the exact detail source-verification caught (the two give different results whenever a weight isn't 1.0, and guessing wrong would have silently produced vectors that don't match the reference Python/Rust output). Normalization uses an epsilon floor so a token-less input yields a zero vector instead of a division by zero. Tests hand-compute the expected pooled vectors for a small synthetic vocabulary and safetensors fixture, including a dedicated test that distinguishes the weighted-sum/token-count-denominator behavior from the (wrong) weighted-sum/sum-of-weights alternative.
similarity(text1, text2): cosine similarity between two pooled embeddings. mostSimilar(text, topK): nearest vocabulary tokens to a pooled query vector, brute-force over the vocabulary (fine at the tens-of- thousands-of-rows scale this module targets; an ANN index is a documented, deferred follow-up, not v1 scope). Excludes the special tokens only; a single-word query's own vocabulary row is, correctly, its own top match, unlike gensim's convention of excluding the query word, which does not generalize to multi-word text queries anyway. analogy(a, b, c, topK): the classic word2vec vector arithmetic (embed(b) - embed(a) + embed(c)), additionally excluding a, b, and c themselves from the results, which is load-bearing here (not just convention) since all three are trivially close to the constructed target vector. Tests use a small fixture with genuinely non-collinear vectors (the pooling-math fixture in StaticEmbeddingModelTest is deliberately collinear, which is ideal for hand-computing weighted averages but would make every pairwise similarity a trivial 1.0), built so the analogy has an exact answer: king - man + woman == queen.
…followup jmh profile Same opt-in jmh Maven profile pattern already used by opennlp-runtime (build-helper adds src/jmh/java as a test source root, jmh-core plus the annotation processor, activated only via -Pjmh; the default mvn verify is unaffected). Fixture is synthesized at the real minishlab/potion-base-8M scale (29,528 rows, 256 dimensions, both verified against the live model repo earlier) rather than downloaded, so the benchmark has no network dependency, but seeded with real English words so the benchmark sentences hit actual vocabulary entries instead of degenerating into all-[UNK] lookups. Forked run (2 forks x 10 iterations, the annotated configuration, not the quick-iteration main() override): embed (5 short sentences/op): 999,222.698 +/- 29,452.980 ops/s mostSimilarTop10 (full ~29.5k-row scan): 3,292.110 +/- 203.639 ops/s This is the JVM-only raw-throughput baseline the design doc calls for before any "faster than Python" claim; the concurrent-load comparison against a Python baseline is a separate, later benchmark.
…ighbor scan, thread-safety hardening
Two analogy() bugs fixed. Passing equal terms crashed with
IllegalArgumentException("duplicate element") from Set.of; and the
exclusion compared raw input strings against vocabulary tokens, so on
an uncased model analogy("Man", "King", "Woman", k) handed "king"
straight back as a result. Exclusion now folds the terms through the
model's own tokenizer and excludes the resulting vocabulary rows,
which makes it case- and accent-consistent with embed() and tolerant
of equal or multiword terms. Both are pinned by new tests.
Nearest-neighbor scan reworked around three observations: per-row L2
norms are constants of the model, so they are precomputed at load
instead of recomputed (with a sqrt) for every row on every query; the
top-K selection now uses a bounded min-heap over primitive parallel
arrays instead of materializing and fully sorting one record per
vocabulary row per query; and the special-token check is a
precomputed boolean mask instead of per-row string hashing. The dot
loop uses four accumulators because the JIT must not reorder
floating-point additions and so cannot unroll the reduction itself.
The zero-norm-row NaN guard is preserved and now has its own test.
embed() drops an OptionalInt allocation per token (primitive -1
sentinel) and hoists the weight branch out of the accumulation loop.
Forked JMH, same configuration and fixture as the recorded baseline:
embed: 999,222 -> 1,041,654 ops/s (+4.2%)
mostSimilarTop10: 3,292 -> 9,173 ops/s (2.79x)
Thread safety reviewed and hardened: @threadsafe on
StaticEmbeddingModel, SafetensorsFile, and WordPieceVocabulary; the
class javadoc now documents why the one piece of global mutable state
in the tokenizer chain (WhitespaceTokenizer.INSTANCE's keepNewLines
flag) cannot affect results, since BERT basic tokenization replaces
all whitespace with plain spaces before that split runs; and a new
concurrency test runs 8 threads against one shared instance comparing
every result to the single-threaded reference.
Replaces the whole-file byte[] (capped at 2 GB by Java's int-indexed arrays, and failing as an opaque OutOfMemoryError beyond it) with positional FileChannel reads: the header is read eagerly, tensor data streams straight into the caller's float[] through a reused 1 MB chunk. File size is now unlimited; the remaining ceiling is per decoded tensor (a float[] holds at most ~2.1 billion elements) and is checked with a clear message. Peak load memory drops since file bytes and the decoded array no longer coexist. A file truncated between read() and readFloat32() fails loud instead of returning partial data.
…rbage Writing the direct tests surfaced one gap: parseTop stopped at the closing brace and silently ignored anything after it. Trailing whitespace stays legal (writers space-pad the header to align the data section), any other trailing content now fails loud.
Distillation, so producing a table no longer needs a Python environment:
- Add ModelDistiller, reproducing Model2Vec in Java: clean the teacher's vocabulary,
run every surviving token through the teacher's ONNX graph as [bos, token, eos] and
mean-pool the last hidden states, project onto the top principal components, then
scale each row by its Zipf weight sif / (sif + p).
- Add OnnxTeacherEncoder for that forward pass, feeding the graph exactly the inputs it
declares (input_ids, attention_mask, plus a zero token_type_ids for the BERT-family
graphs that ask for one), and declare the onnxruntime dependency in the module at the
root-managed version, the same engine opennlp-dl already runs.
- Add TeacherTokenizer, which decides which vocabulary rows survive into the table and
rewrites tokenizer.json to describe it, copying every field it does not change byte
for byte so the result stays a faithful fast-tokenizer description.
- Add RandomizedPca (Halko, Martinsson, Tropp), because a dense SVD of a 250k-row
multilingual matrix is not practical in pure Java; component signs are fixed the way
scikit-learn's svd_flip fixes them, so two runs are comparable rather than mirrored.
- Add SafetensorsWriter, the write side of the format SafetensorsFile reads, streaming
the matrix in chunks so its overhead beyond the caller's array is constant.
- Add HuggingFaceModelCache so -teacher accepts a hub id and its files download once
into a local cache; an optional file the repository does not have (a WordPiece
teacher has no SentencePiece model) is reported absent, not an error.
- Register the DistillModel tool in the module CLI. It ends by assembling and verifying
its own output directory, so a run that prints a summary is a directory that loads.
Documentation:
- Rewrite TRAINING.md around the DistillModel command instead of the Python uv and
model2vec setup, keep the Python flow only as the parity reference, and record that
two tables distilled independently from one teacher agree on their pairwise geometry
but not axis by axis.
- Point the README "Getting a model" section at DistillModel as an alternative to
downloading an already released table.
Folded duplication:
- Move firstRegularFile into ModelFileNames, next to the name lists it scans;
StaticEmbeddingModel and ModelAssembler each carried a copy.
- Fold the boolean and string readers of FlatJsonFields onto one top-level walker with
a ValueReader seam, so the object grammar, the duplicate-field check, and the
trailing-content check exist once.
- Collapse the two identical file checks in EmbeddingVocabulary into requireRegularFile.
- Reduce SentenceVectorsDL.declaredOutputDimension to the single first-output read the
loop actually performed, since every path returned on the first output anyway.
Javadoc and commentary:
- Document JsonCursor.consumeLiteral, requireEnd, and the new position(), and the two
writer overloads in SafetensorsTestFiles.
- Fix TensorInfo's stale link to SafetensorsFile.readFloat32, which is readFloats now,
and move its element-count description into the {@return} form.
- Describe Neighbor.token as one subword piece of the model's tokenizer rather than a
WordPiece, now that SentencePiece tables load too.
- State on TextEmbedder that thread safety is implementation specific, which is what
the two implementations actually promise, and mark up null in its tags.
- Drop the commentary that narrates history rather than behavior: the benchmark's
design-doc justification, the "before the fix" and "used to crash" notes in the
similarity tests, and the "untouched by the interface adoption" note in the DL test.
Tests:
- Add EmbeddingTestFixtures holding the king/queen/man/woman analogy table and the JSON
string quoter, and point the similarity, concurrency, SentencePiece, usage-example,
and assembler tests at it instead of their own copies of each.
- Build the size-mismatch and zero-vector fixtures with SafetensorsTestFiles rather
than hand-rolling the header and the little-endian payload in the test.
- Pin FlatJsonFields.topLevelString: escapes, absent versus explicit null, nested names
not matching, non-string values, duplicates, and null arguments.
- Add ModelDistillerTest, RandomizedPcaTest, and TeacherTokenizerTest over the
distiller's pure pieces, and drop a duplicate IOException import in
FlatJsonFieldsTest.
Build:
- Sort opennlp-spellcheck before opennlp-subword in the extensions module list.
…urface A review of the model distiller, which had not been reviewed before, found four defects that produce a wrong result rather than an error, and a set of classes with no tests at all. Correctness: - RandomizedPca floored the CholeskyQR jitter at an absolute value, so the decomposition was not scale invariant. Scaling the input down by 1e-6 turned an exact rank-6 recovery into an explained variance of 0.0017 with pairwise dot products wrong by three orders of magnitude, while still returning a plausible table. The jitter is now relative to the Gram trace. - RandomizedPca returned NaN instead of failing when the centered matrix was exactly zero, and a single non-finite input value poisoned its column mean and turned the whole result NaN. Both are now rejected. - The distiller guard named nanToZero tested only Float.isNaN, so an infinity from the teacher passed through into the PCA. Renamed to nonFiniteToZero and switched to Float.isFinite, with the divergence from numpy nan_to_num noted. - The PCA-skip branch kept the un-reduced matrix but left the requested dimension as the row stride, so the Zipf loop scaled the wrong cells and the writer rejected the array. Reachable with any vocabulary smaller than pcaDims, for example 100 tokens at the default 256. Tests, all previously absent: - SafetensorsWriter had no test file and its one case wrote six floats, so the 1 MiB streaming loop never crossed a chunk boundary. Added a 400x1024 round trip along with the header layout, alignment and reject paths. - ModelDistillerTest never called distill, leaving every argument check unverified. It now covers all five reject paths. - Added first tests for ModelFileNames, HuggingFaceModelCache, OnnxTeacherEncoder and the module CLI. - RandomizedPcaTest pins scale invariance, the non-finite rejection and determinism across pool sizes. Also corrected the TRAINING.md WordPiece section, which told the reader to run AssembleModel on output the distiller has already assembled. Module tests go from 156 to 273, none skipped.
The distiller fetched teacher weights from a moving ref and executed the ONNX graph without checking anything. DownloadUtil has always refused a model whose sha512 sidecar it cannot read, so this brings the hub path to the same posture. - Resolve the ref to a commit once, then request every file at that sha, so a force-push midway cannot mix two revisions into one cache directory. A teacher may now name a revision as org/model@revision. - Verify every file against the digest the hub publishes in x-linked-etag, choosing the algorithm by hex length: 40 is the git blob SHA-1 over "blob <len>\0" and the bytes, 64 is the SHA-256 of the content. The digest is computed over what was written to disk. A mismatch deletes the partial file and throws, in DownloadUtil's wording. - Refuse a file whose digest the hub does not publish, rather than accept something unverifiable. This is the point of the change. - Record the resolved commit in .opennlp-revision, which also marks the directory as a complete snapshot, and carry it into the distilled model's config.json as teacher_revision so a table can name the teacher it came from. - Make the cache directory name injective. It replaced '/', '.' and '@' without distinguishing them, so acme/model@v1, acme/model.v1 and acme/model_v1 shared one directory, and the cached path answers without contacting the hub, which would have served one teacher another's files. Headers are read by walking HttpResponse.previousResponse(): with Redirect.NORMAL the final CDN response carries neither header, while the redirecting hub response carries both. A test fails if that walk is removed. Tests need no network. A loopback HttpServer serves canned replies and records the requests made, covering both digest forms, a corrupted body, a missing and six malformed etags, optional and required 404s, the redirect path, the zero request cached path, revision pinning and the directory collision. Module tests go from 273 to 310, none skipped.
The default TextEmbedder.embedAll embeds one text at a time, so a document pipeline paid one session run per sentence or token span. SentenceVectorsDL now tokenizes the batch up front, groups inputs by tokenized length, and runs each group through the session once with shape [group size, length]. Grouping by length means a batch never pads: every row is computed from exactly the tensors its single-input call would have used, so results match the per-text calls bit for bit, which the new test pins with exact float equality over mixed-length inputs.
Annotate the public embeddings types with @experimental, note that in the manual, and fail loud on a lone "-" in a skipped safetensors field.
…nd edge tests Annotate ModelAssembler, ModelDistiller, and TextEmbedder with @experimental like the other public embeddings types and extend the manual's note. Link Model2Vec and the randomized SVD paper at first mention. Make the usage example test mirror the manual's directory-load listing. Cover whitespace-only and supplementary-plane input in the embed path. Document the Checksum constructor.
Malformed model content (malformed safetensors headers and config files, dimension and row-count disagreements, ambiguous matrix sources, bare minus headers) now throws the checked opennlp.tools.util .InvalidFormatException across StaticEmbeddingModel.load and every loader it calls, matching SentencePieceTokenizer.load. IllegalArgumentException stays for caller argument errors only. The CLI tools report the new type with the same exit code as before.
…rivate readFloat32(String) and metadata() have no main-source callers; only the module's own tests use them. Shrink the experimental public surface.
…e manual Add a Command Line Tools section to the embeddings chapter covering the bin/embeddings launcher and both tools: purpose, invocation shape, and the load-based verification each run ends with, in the style of the manual's other tool sections.
The distiller writes the pooling field but the loader never read it, so a third-party model declaring another pooling silently mean-pooled. Only mean pooling is implemented, so the loader now rejects any other declared value with an InvalidFormatException naming it; a declared mean still loads.
mostSimilar and analogy sized the candidate arrays by the raw topK before any clamping, so mostSimilar(word, Integer.MAX_VALUE) failed with an OutOfMemoryError. The scan can never yield more than one neighbor per row, so the capacity is now the smaller of topK and the row count.
The distiller path zeroes non-finite teacher values, but the loader accepted any bytes, and a single NaN row defeats the zero-norm guard and the TopK comparison, silently corrupting similarity rankings. Loading a matrix that holds a NaN or infinity now throws InvalidFormatException naming the row.
do_lower_case=false was never tested; the new case asserts that a cased model matches lower-case vocabulary entries as-is and folds upper-case text to the skipped unknown token instead of lower-casing it first.
The chapter's second programlisting (the explicit load and loadSentencePiece overloads) had no test pinning it. testExplicitOverloads loads both fixture layouts exactly as the listing shows, backed by a new minimal SentencePiece directory fixture, and the chapter now cites the test the way it already cites the directory-load listing.
…test The headline use case, ranking documents against a query by cosine similarity, existed only in the module README. The chapter gains a Semantic Search section whose listing scores each document with the public similarity method and sorts by descending score, and the new StaticEmbeddingSearchExampleTest asserts the exact ranking on the analogy fixture's known geometry.
The hub cache compiled two Patterns for the org/model@revision reference and the hex shape of commit shas and digests. Hand scans over the same ASCII grammars replace them; the module's parsing is now regex-free throughout.
…d pieces A distillation can now take a term list, a learned corpus vocabulary of whole words and multi-word phrases. Each term is segmented by the teacher's own tokenizer, encoded through the teacher as one sequence, and appended to the table after the subword rows, recorded as terms.txt in the model directory. The same PCA and Zipf pipeline spans all rows, and a term equal to a surviving vocabulary token is dropped as a duplicate row. At embed time the model matches text against its terms greedily longest-first over case-folded word runs (StringUtil.toLowerCase, one code point to one code point) and pools a matched term's single row in place of its words' subword pieces; text between matches tokenizes as before, and a model without a terms file embeds exactly as it did. Terms are neighbor candidates in mostSimilar like any token. The DistillModel tool gains -terms, documented in the manual. TeacherTokenizer's two Patterns (the unused-token filter and the template splitter) are replaced with cursor scans along the way.
…-Max grids, packed codes
… the QuantizeModel tool
…t quantization, expand tests Fail loud when a directory holds both model.quantized and model.safetensors instead of preferring one silently. Widen the QuantizeModel seed argument to long so the full seed space is expressible. Add a Quantized Models manual section pointing at the workflow tests. Convert reconstruction and embed-parity tests to parameterized bit-width cases, add the SentencePiece quantized path and a both-files-present rejection, and share the cosine and SentencePiece fixture helpers.
A quantized file header declaring a dimension near 2^29 made paddedDimension * bits overflow a signed int, so the per-row byte count went negative and reading the file crashed with an undocumented NegativeArraySizeException instead of a clean rejection; the same overflow would corrupt the bit addressing in readCode/writeCode. Compute the row byte count in long arithmetic through a single range-checked helper used by quantize, read, and the constructor. Add edge-case tests across non-power-of-two dimensions, a one-dimensional matrix, and adversarial rows the rotation must still reconstruct.
… matrices (failing tests)
…eption Content errors in read now throw the checked loader exception instead of IllegalArgumentException, with a size plausibility guard before any allocation. The ambiguous matrix source and the quantizer's missing safetensors follow the same contract. The CLI pin includes QuantizeModel, which this branch registers.
…d table path The row count disagreement and the wordpiece unknown-token check now throw InvalidFormatException like the rest of the module after the OPENNLP-1877 conversion.
…path and a size bound before allocation (failing tests) The loader contract says malformed file content fails with the checked InvalidFormatException, but the decoded-norm, pooling-weight, stored-grid, and trailing-byte rejections still throw IllegalArgumentException, and a small hostile file declaring huge dimensions reaches per-row allocation and dies with EOFException instead of a format error. These tests pin the intended behavior and fail against the current reader: - testReadRejectsForeignAndTruncatedFiles now expects InvalidFormatException for trailing bytes (was pinning IllegalArgumentException, the wrong contract) - testDeclaredPayloadBeyondFileSizeFailsBeforeAllocating: a 1.1 MB file declaring 1,000,000 rows of 512 dims at 4 bits must fail fast, before allocating 256 MB of codes plus 8 MB of scales and norms - non-finite stored grid levels, decoded norms, and pooling weights, and a pooling-weight flag with no weights present, must all fail with InvalidFormatException
…eption and bound declared sizes before allocating read(Path) promises the checked InvalidFormatException for malformed content, but four rejections still threw IllegalArgumentException and escaped every catch (IOException): an invalid decoded norm, a non-finite pooling weight, trailing bytes after the declared content, and the row-byte-count and storable-size checks reached from a hostile header. A stored grid that fromLevels rejects also surfaced as IllegalArgumentException. All of these now throw InvalidFormatException naming the file and the offending field; constructor validation for programmatic callers stays IllegalArgumentException. The reader also only checked rowCount and dimension individually against the file size, so a 1.1 MB file declaring 1,000,000 rows of 512 dims at 4 bits forced a 256 MB code allocation before any content check. The header fully determines the file size, so the declared total (28 fixed bytes, the grid levels, a scale and a decoded norm per row, the flag byte, and the packed codes, plus the per-row pooling weights when flagged) is now held against the actual file size and rejected before anything row-sized is allocated.
…ding The class comment and the manual claimed a 500,000-row, 300-dimension table drops to 77 MB at 4 bits. The Hadamard rotation pads rows to the next power of two, so 300 dimensions store 512 codes per row, and each row also carries two floats (the fitted scale and the decoded norm), not one. Measured from a written file, that is 264 bytes per row: 132 MB against 600 MB of float32, 4.5 times smaller, not 7.8. The overall shrink range becomes roughly 4 to 16 times depending on bit width and padding distance. The padding behavior itself is unchanged.
This was referenced Aug 16, 2026
krickert
added a commit
that referenced
this pull request
Aug 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Depends on #1152 (OPENNLP-1877 static embeddings). Please review this after its parent.
This adds compact quantized storage and similarity search for OpenNLP static embedding models:
The implementation remains inside the existing opennlp-embeddings module and does not introduce a general search server or distributed index.
Verification
The branch was restacked on the current #1152 tip and passed the complete 15-project compilation, packaging, Checkstyle, forbidden-API, Javadoc, and RAT reactor gate with opennlp.forkCount=1. The quantization and direct embeddings suites also passed on the complete stack.
The unrestricted upstream reactor test command additionally encountered unrelated model-download failures in inherited runtime and formats tests. Those failures could not obtain public tokenizer and sentence models; 1,682 runtime unit tests passed before the download-dependent failures.
JIRA
https://issues.apache.org/jira/browse/OPENNLP-1895