Add grobid_custom_hybrid profile with SciELO-preprints reference segmenter - #680
Draft
de-code wants to merge 22 commits into
Draft
Add grobid_custom_hybrid profile with SciELO-preprints reference segmenter#680de-code wants to merge 22 commits into
de-code wants to merge 22 commits into
Conversation
…enter Introduces a new sequence-model profile that overlays the grobid 0.9.0 CRF/wapiti baseline with our own retrained DELFT models, starting with a CustomBidLSTM_CRF reference segmenter trained on SciELO preprints. Intended to grow as more custom models are retrained and swapped in.
ScienceBeam Parser EvaluationOverall (277 docs across 6 corpora)grobid 0.9.0-crf (default): 277 docs | sciencebeam-parser:main-d0f4519b-20260817.1058 (grobid_crf): 277 docs | sciencebeam-parser:pr-680-5716df43-20260817.1120 (grobid_custom_hybrid): 277 docs
biorxiv (43 docs)grobid 0.9.0-crf (default): 43 docs | sciencebeam-parser:main-d0f4519b-20260817.1058 (grobid_crf): 43 docs | sciencebeam-parser:pr-680-5716df43-20260817.1120 (grobid_custom_hybrid): 43 docs
ore (50 docs)grobid 0.9.0-crf (default): 50 docs | sciencebeam-parser:main-d0f4519b-20260817.1058 (grobid_crf): 50 docs | sciencebeam-parser:pr-680-5716df43-20260817.1120 (grobid_custom_hybrid): 50 docs
pkp (50 docs)grobid 0.9.0-crf (default): 50 docs | sciencebeam-parser:main-d0f4519b-20260817.1058 (grobid_crf): 50 docs | sciencebeam-parser:pr-680-5716df43-20260817.1120 (grobid_custom_hybrid): 50 docs
scielo_br (50 docs)grobid 0.9.0-crf (default): 50 docs | sciencebeam-parser:main-d0f4519b-20260817.1058 (grobid_crf): 50 docs | sciencebeam-parser:pr-680-5716df43-20260817.1120 (grobid_custom_hybrid): 50 docs
scielo_mx (34 docs)grobid 0.9.0-crf (default): 34 docs | sciencebeam-parser:main-d0f4519b-20260817.1058 (grobid_crf): 34 docs | sciencebeam-parser:pr-680-5716df43-20260817.1120 (grobid_custom_hybrid): 34 docs
scielo_preprints-jats (50 docs)grobid 0.9.0-crf (default): 50 docs | sciencebeam-parser:main-d0f4519b-20260817.1058 (grobid_crf): 50 docs | sciencebeam-parser:pr-680-5716df43-20260817.1120 (grobid_custom_hybrid): 50 docs
|
…enter Introduces a new sequence-model profile that overlays the grobid 0.9.0 CRF/wapiti baseline with our own retrained DELFT models, starting with a CustomBidLSTM_CRF reference segmenter trained on SciELO preprints. Intended to grow as more custom models are retrained and swapped in. Also wires up the profile:grobid_custom_hybrid PR label in the benchmark workflow so it selects this profile.
Replaces the hardcoded per-profile contains() checks with a step that extracts any profile:<name> label directly, validated against a safe character set, so new profiles no longer require a benchmark.yml edit.
Authenticate CI to GCP via Workload Identity Federation so the grobid_custom_hybrid profile can pull the reference-segmenter model from gs://sciencebeam-v2-models while the trained model's release is not yet public.
TensorFlow's built-in gs:// filesystem support only recognizes plain service-account JSON keys, not the Workload Identity Federation external_account credential files google-github-actions/auth produces - those reference a credential source that can't be resolved from inside the sciencebeam-parser container, causing anonymous (unauthenticated) requests. Decode a service account key from the GCS_MODEL_READER_SA_KEY_B64 secret instead and mount it into the container.
Points the citation model at the new CustomBidLSTM_CRF model in the private sciencebeam-v2-models GCS bucket, replacing the grobid 0.9.0 wapiti model, while its release is not yet public.
Collaborator
Author
|
This doesn't look that good unfortunately |
Custom hybrid profiles are expected to grow beyond this one, so inherit from a base profile via a code-resolved `extends` key rather than a YAML anchor - it deep-merges instead of replacing whole model entries, gives clear errors for unknown/circular references, and keeps working if profiles ever get split across files.
…file One conflict, in .github/workflows/benchmark.yml. This branch had already replaced the job-level BENCHMARK_PROFILE env expression with the "Resolve benchmark profile" step, which accepts any profile: label and validates it; main meanwhile added BENCHMARK_CORPORA for the opt-in PLOS corpus. The resolution keeps the step and drops main's BENCHMARK_PROFILE expression, which the step supersedes, and takes BENCHMARK_CORPORA unchanged.
… Sciety Preprints and ORE
…file One conflict again, in .github/workflows/benchmark.yml, the same one as the previous merge: main still carries the job-level BENCHMARK_PROFILE expression that this branch replaced with the "Resolve benchmark profile" step, so the resolution drops it once more. #691 also rewrote the comment above BENCHMARK_CORPORA, whose body had already auto-merged with the new github.ref == 'refs/heads/main' condition, so that comment is taken from main.
The step wrote BENCHMARK_PROFILE to the environment and said nothing, so which profile a run used could only be inferred later, from the parser invocation further down the log. It now reports the profile and where it came from - label, workflow_dispatch input, or the default. A notice rather than a plain echo, so it also surfaces at the top of the run.
The labels expression renders as null on push and workflow_dispatch, where there is no pull request, and plain .[] treats iterating that as an error. It was masked: jq's failure sits mid-pipeline, so head's success is what the assignment saw, and the empty result fell through to the default. The log still carried an error line on every main run, and setting shell: bash would have added pipefail and turned it into a failed step - on exactly the runs that produce the recorded baseline. .[]? yields nothing instead.
Collaborator
Author
|
Still showing a regression with those retrained models. That indicates issues with the training data (or how it is being combined) |
The visited profiles were carried as a frozenset, so the chain the error advertises printed in arbitrary order - and only across processes, since within one it is stable, which is how it passed a test suite. A tuple keeps the order, and the new test pins it. Matches the standalone extends branch exactly, so config.py does not conflict when this branch next merges main.
…file Three conflicts, each from this branch having a fuller version of what main just gained. benchmark.yml: main's profile-resolution step arrived where this branch already has the GCS key step in front of it, so the resolution keeps both, GCS first. config.yml: main's grobid_custom_hybrid states only `extends`, this branch overrides citation and reference_segmenter on top of it, so the overrides stay. config_test.py takes main's DEFAULT_CONFIG_FILE import, and its assertion is rewritten. It asserted the hybrid resolves to exactly what grobid_crf resolves to, which is true on main and false here - which is what it was written to catch. It now names what this branch actually intends: the same ten models, differing in citation and reference_segmenter alone.
It named which models the profile overrides, so a config change had to come with a test change - and it was the config, not any behaviour, that the assertion described. That extends inherits the base and lets a profile override one model is already covered by profile_b_extended in MINIMAL_PROFILE_CONFIG, on a fixture that no model swap disturbs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
part of https://github.com/eLifePathways/ScienceBeam2.0/issues/113
Introduces a new sequence-model profile that overlays the grobid 0.9.0 CRF/wapiti baseline with our own retrained DELFT models, starting with a CustomBidLSTM_CRF reference segmenter trained on SciELO preprints. Intended to grow as more custom models are retrained and swapped in.