Support resume and purge modes for index recovery

[DylanWelzel]

Summary

This PR adds first-class index recovery modes so a hub or frontend caller can recover from an interrupted or failed indexing job without having to manually clean up Elasticsearch state.

The branch has been updated with the latest 1.1.x changes and this PR now targets 1.1.x.

The new supported modes are exposed by index_manager and passed through the existing /index command:

• index: create a fresh index and fail if it already exists, preserving the existing default behavior.
• resume: keep an existing index and index only documents that are missing from Elasticsearch.
• purge: delete an existing index first, then recreate and index from scratch.
• merge: preserve the existing merge behavior for hot/cold indexing flows.

Why This Is Needed

The frontend needs to give operators a safe choice when a full index already exists or when a previous indexing run failed partway through.

Before this change, the default index path intentionally failed if the target index already existed. That is the right default for avoiding accidental overwrite, but it made recovery awkward:

• A partially completed index could not be resumed from the UI.
• Rerunning the same full release often failed immediately because the Elasticsearch index already existed.
• Operators had to know whether to manually delete the index or run a hub command with a non-default mode.
• The frontend had no machine-readable list of supported index modes or descriptions to explain the choices.

This PR makes the recovery behavior explicit in the backend API so the frontend can present a clear resume vs purge decision when an index exists.

What Changed

Index mode metadata

IndexManager.index_info() now includes an index_modes dictionary describing the supported modes. This gives callers a stable backend-provided contract instead of hardcoding all mode labels and descriptions in the UI.

Job state context

The index job state registrar now accepts extra context when a step starts. The pre-index step records the requested mode when the mode is not the default index mode.

This helps build logs and UI state explain whether a recovery attempt was a resume, purge, or other non-default indexing operation.

Pre-index behavior

The existing pre-index mode handling is preserved and made easier for callers to discover:

• index requires the target index not to exist.
• resume and merge require the target index to already exist and skip index creation.
• purge deletes the target index if present, then creates a new one.

Resume implementation

The resume path now checks which IDs are already present in Elasticsearch and indexes only the missing IDs.

The existence check is batched so large indexing batches do not issue one huge IDs query or exceed common Elasticsearch search window constraints. The shared _index_ids() helper also avoids mutating self.ids during resume.

Client cleanup

Index, merge, and resume tasks now close their Elasticsearch and Mongo clients in finally blocks. This keeps process workers from holding unnecessary connections after a batch finishes or fails.

Index lookup compatibility

IndexManager.get_indexes_by_name() now accepts either _meta.stats.total or _meta.stats.total_documents when building /indexes_by_name results.

This fixes a compatibility issue seen after a purge/reindex run where Elasticsearch contained the newly created index, but /indexes_by_name returned no result because the index metadata only had total_documents. Without this fallback, the frontend could not rediscover the new index in dropdown lookups even though the index was present and populated in Elasticsearch.

User Impact

Operators can now safely recover indexing from the frontend or API:

• Choose resume when an index partially completed and mappings/settings are still valid.
• Choose purge when a clean rebuild is safer because mappings/settings changed or the index may be inconsistent.

Existing callers that do not pass a mode continue to use the default index behavior.

The /indexes_by_name endpoint now also remains compatible with indexes whose metadata stores the count as total_documents, so freshly purged/reindexed indexes continue to appear in frontend index selectors.

Validation

• Merged origin/1.1.x into resume-purge-logic without conflicts.
• Confirmed the failing scenario on su12: Elasticsearch had disease_20260331_wentol7t, while /indexes_by_name?index_name=disease_20260331_wentol7t&env_name=su12_es8&limit=1 returned an empty result because the metadata used stats.total_documents.
• python3 -m py_compile biothings/hub/dataindex/indexer.py biothings/hub/dataindex/indexer_registrar.py biothings/hub/dataindex/indexer_task.py
• git diff --check origin/1.1.x...resume-purge-logic

Notes

This PR is intentionally scoped to the backend contract and indexing behavior. The frontend can consume the existing /index endpoint with mode: "resume" or mode: "purge" and can read mode descriptions from /index_manager.