Switch Solr from Cloud mode to standalone mode for self-contained backups#249
Draft
gaurav wants to merge 1 commit into
Draft
Switch Solr from Cloud mode to standalone mode for self-contained backups#249gaurav wants to merge 1 commit into
gaurav wants to merge 1 commit into
Conversation
…kups Remove ZooKeeper/cloud mode (-DzkRun) so the schema lives alongside the Lucene index in the core directory, making the backup tarball fully self-contained. Restore is now: extract tarball + docker compose up. Key changes: - docker-compose.yml: remove `command: ['-DzkRun']` (standalone by default) - data-loading/setup_solr.sh: replace Collections API with Core Admin API (configSet=_default); fix JSON syntax error (missing comma in types field) - data-loading/Makefile: remove -cloud flag; replace replication backup (start-solr-backup / check-solr-backup) with filesystem tar of name_lookup/ core directory (solr-data.tar.gz), with read-only solrconfig installed first - data-loading/Dockerfile: remove -cloud flag; COPY conf/ for read-only config - data-loading/conf/solrconfig.xml: new — NoOpUpdateHandler (read-only) + ManagedIndexSchemaFactory; installed into core before tarring - data-loading/conf/managed-schema.xml: new — reference schema documentation - solr-restore/: deleted — no longer needed; tarball is self-contained - documentation/Deployment.md: updated storage estimate, extraction step, removed restore script steps - data-loading/README.md: updated Makefile section and step-by-step instructions Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2 tasks
gaurav
added a commit
that referenced
this pull request
Jul 23, 2026
…278) ## What & why Building a NameRes instance loads ~130 GB of Babel synonyms into Solr and ships a backup a worker restores. The old pipeline had three problems: a slow serial load (per-file hard commit + `sleep 60`), a Solr schema **defined twice** (loader *and* the Helm restore script) that had already drifted, and ZooKeeper running for what is a one-shard, one-node job. This PR reworks the whole flow around a **self-contained standalone core**: the backup is a whole Solr core — config + schema + index — so restoring it is just "untar into the Solr home and start Solr". No ZooKeeper, no collection creation, no restore-time schema setup. This supersedes #249. ## Key changes - **Single source of truth for the schema:** new checked-in configset at `data-loading/configsets/name_lookup/conf` — two files, ~220 lines, **written for NameRes rather than copied from Solr's `_default`**. Field types are carried over verbatim so analysis behaviour is unchanged, the malformed `types` field is fixed, the index buffer is larger, `autoSoftCommit` is off and `queryResultCache` is grown. Everything in it is a decision we made, so there is nothing to reconcile against upstream on a Solr upgrade; the copy it replaced was 44 files, 70 field types and 69 dynamic fields, of which NameRes uses 6 field types and 13 fields (42 of the files were language stopword lists, which Babel synonyms can never use). The schema is now **immutable** (`mutable=false`), and schemaless mode is off in `solrconfig.xml` rather than in `configoverlay.json`, which Solr's Config API rewrites — so no file in the configset is one Solr can rewrite behind us. An unexpected Babel key is now a hard `unknown field` error instead of a silently invented field. - **Fast, guarded load:** `setup-and-load-solr.sh` streams files to Solr in parallel with a single deferred commit (no per-file commit, no sleeps). Because a parallel load of a huge dataset is scary, it **counts input documents before loading and compares the increase in Solr's count afterward**, with `curl --fail` on every upload — a drop or a bad file aborts with a non-zero exit. The count is the load-bearing half: Solr answers **200 and indexes nothing** for a file of plain text, so `--fail` alone would let it through. - **Standalone everywhere:** loader `Dockerfile`, `docker-compose.yml`, and CI drop cloud mode; Solr bumped to 9.10. The Makefile creates the core from the configset, optimizes the index, and tars the core into `snapshot.backup.tar.gz` (with `pigz` when available — single-threaded gzip on an already-compressed 130 GB index was hours for a few percent). - **`/status` fixed for standalone:** it looked up the SolrCloud core name `name_lookup_shard1_replica_n1`, so under standalone it reported `Expected core not found.` with no document counts — quietly, since the endpoint still answers 200 and the Helm probes only check that. The core now comes from `SOLR_CORE` (default `name_lookup`), with a fallback to the single core present so old cloud-built backups keep working. - **Version pinning by role:** the builder names an exact release tarball (`SOLR_VERSION=9.10.0`) and is the floor for anything serving the backup; the servers (docker-compose, CI, Helm chart) track the `9.10` line, so they pick up patch releases and stay at or above the builder automatically. - **Deleted:** `setup_solr.sh`, `solr-restore/` and `configoverlay.json` (all obsolete). - **Load sized to the container, not the node:** parallelism and `pigz` now come from the cgroup CPU quota rather than `nproc`, which inside a pod reports the node's cores — 8 CPUs of quota on a 64-core node used to mean 64 throttled uploads. The loading heap drops from 220G (`solr -m` commits both `-Xms` and `-Xmx`) to 31G, since indexing wants page cache rather than heap, and `ramBufferSizeMB` rises to 2G — the buffer is a budget shared across indexing threads, so at 32 parallel uploads that is ~64 MB per segment instead of ~16 MB, and four times fewer segments to merge. - **Kubernetes loading job reworked:** the pod goes to 32 CPUs / 128Gi, and `/var/solr` is sized at 600Gi (the optimize writes the new segment before dropping the old ones, so it peaks at ~2x the finished index). Moving the index onto a node-local **NVMe ephemeral volume** is the change that would actually speed this up — it takes the whole write load of the indexing run and the optimize, and it is the one volume we could afford to lose — but it is blocked on #280 (the namespace cannot create `nvme-ephemeral` volumes), so **both volumes are persistent PVCs today** and both need deleting by hand. Makefile stamp files now live on the volume whose state they describe, so they disappear with the index if that volume is ever made ephemeral. New [`data-loading/kubernetes/README.md`](data-loading/kubernetes/README.md) covers the volume split, sizing, the Grafana panels that answer “would more of this help”, and every knob. - **CI image builds cached:** the data-loading image was rebuilt from scratch every run — `apt upgrade`, seven `apt install`s and a ~250MB Solr tarball — taking ~25 minutes. It now uses the GitHub Actions cache. - **Helm chart** (in `translator-devops`, edited copy under `data/name-lookup/`): standalone Solr, idempotent `download.sh`, `restore.sh` gutted to blocklist-only, heap lowered for page cache. ## Validation CI now runs the whole thing end to end, against a Solr loaded by the new pipeline: - **`pytest`: 28/28 pass** (including `test_status`, which the standalone switch initially broke). - **Configset verified against a real Solr:** core creates from the two files, load succeeds (so the `uuid` processor and `/query` survived the trim), an unknown field is rejected with a 400, a Schema API write is refused with `schema is not editable`, and highlighting still returns spans. - **The loader must abort** on a file that does not load — the guard is what stands between us and a silently half-loaded index, so it is tested rather than assumed. - **The loader must refuse a core that already has documents.** Loading assigns fresh UUIDs, so a re-run over the same files doubles the index, and the count guard cannot see it — the delta is still exactly right. This is the one way the loader can return a corrupt index while reporting success, so it is a hard error (`LOAD_APPEND=1` to override). - **Full backup roundtrip in CI:** tar the core → drop it into a fresh Solr → 89 documents served with **zero schema setup and no `chown`**. The roundtrip deliberately chowns the source core to uid 1000 first, standing in for the loading image, so it proves the tarball's ownership stamping rather than hiding it behind a fixup. Also checked by hand: an unexpected field is rejected with a 400 (schemaless really is off), filenames with spaces and blank lines are counted correctly, and `make data/backup.done` on a restarted container restarts Solr instead of failing — without re-running `solr create` on a core that already exists. Also confirmed the failure the ownership fix prevents: a backup tarred without it restores into a core that will not load, with `SolrException: /var/solr/data/name_lookup/data/index/write.lock` — Solr takes a write lock on the core it serves, and the loading image's uid (1000) is not the serving image's (8983). ## Issues Closes #238, closes #185, closes #256, closes #266. Addresses helxplatform/translator-devops#609. Progresses #265 (heap knob + smaller index; final heap/GC tuning and #272 still need load testing). Out of scope: #267, running natively on SLURM/HPC. ## Pre-merge cleanup — done - [x] **Deleted the `pull_request:` trigger** from `.github/workflows/release-nameres-loading.yml` (c563ded). It only existed to publish a `pr-278` image so a load could run against this branch without merging. - [x] **Reverted `nameres-loading.k8s.yaml` to `image: …:latest`** (b6458cc). Caveat carried over: the publish workflow only tags `latest` on a published *release*, so `latest` becomes this pipeline only once a release is cut — run the next load after this is merged and released. ## The load that already ran — done A full Babel 2026jul22 load (~111Gi index, ~331M docs) was built with the `pr-278` image and is now restored and serving on the cluster, so the workarounds this section used to list are spent. That run predated the review fixes, so the backup's ownership was stamped by hand — the merged Makefile now does it automatically with `tar --owner`. The load ran to completion (no partial-load restart, no stray files to skip). Two problems surfaced during it, both now fixed: the backup's uid ownership (in this branch) and the serving chart's Solr image tag being read as `9.1` (see the reviewer note). ## Note for reviewers The `data/name-lookup/` Helm chart is gitignored here (it's a working copy); its edits live in `translator-devops`. One caveat found while bringing the load up: the Solr image tag must be **quoted** (`tag: "9.10"`) — unquoted, YAML parses `9.10` as the float `9.1` and pulls `solr:9.1`, which cannot init a 9.10-built core (it fails with `_version_ … not searchable`). That fix still needs applying in `translator-devops`. Point `dataUrl` at a backup built by the new loader — old backups won't restore under the new flow. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #238. Everything else is currently written by Claude:
Remove ZooKeeper/cloud mode (-DzkRun) so the schema lives alongside the Lucene index in the core directory, making the backup tarball fully self-contained. Restore is now: extract tarball + docker compose up.
Key changes:
command: ['-DzkRun'](standalone by default)