Summary
Add a multimodal (vision + text) dataset-backed environment: the agent is
shown an image of a mathematical expression and must return its LaTeX
source. Reward is computed server-side against the hidden ground-truth LaTeX.
This environment is built specifically on top of the newly introduced Task
API (#726) and adds dataset streaming for datasets too large to
materialize — the two capabilities this proposal is meant to exercise and
showcase end-to-end.
Why this env
- Exercises the new Task API — a real, dataset-backed consumer of
list_splits / num_tasks / get_task / get_task_range, so the Task API
has a concrete reference implementation beyond the importer templates.
- Dataset streaming — a first environment that streams a Hugging Face
dataset instead of downloading it, so TB-scale datasets (millions of rows,
hundreds of GB of images) can be served from a small Space.
- Multimodal — image input + text (LaTeX) output, gradeable with any
vision-LLM policy. Fills a gap: most reference envs are text-only.
Proposed design
- Episode (bandit):
reset(split, index) → observation with base64 image
(target hidden) → step(LatexOCRAction(latex=…)) → reward, done=True,
ground truth revealed.
- Task API modes (
LATEX_OCR_MODE):
materialize — split loaded + indexed; reset(split, index) random access.
Best when the dataset fits on disk.
stream — sequential cursor over a streamed split, no full download;
observations carry progress (index / total / remaining / pct_done);
num_tasks comes from dataset metadata only; no-repeat within a pass.
reset(index=i) random access is intentionally unsupported (streaming is
sequential). This is the mode for very large / TB-scale datasets.
- Reward (
LatexOCRRubric): 0.8·(1−CER) + 0.2·exact_match, whitespace-
insensitive normalized character edit distance; pure-Python (no extra deps).
- Custom Gradio "Try it" tab: get a task image, type LaTeX, see the score.
- Any
(image, latex) dataset works via LATEX_OCR_DATASET (+ column env vars);
default is unsloth/LaTeX_OCR.
Scope / non-goals
- No new core APIs — uses the existing
Environment + the new Task API as-is.
- Semantic/rendered reward (render-and-compare) is out of scope for v1.
Validation
Verified end-to-end locally and on a deployed HF Space: Task API, streaming
cursor + live progress, no-repeat, and a real vision-LLM policy via the HF
Inference Router (mean reward ~0.77, incl. exact matches). Rubric has unit tests.
Summary
Add a multimodal (vision + text) dataset-backed environment: the agent is
shown an image of a mathematical expression and must return its LaTeX
source. Reward is computed server-side against the hidden ground-truth LaTeX.
This environment is built specifically on top of the newly introduced Task
API (#726) and adds dataset streaming for datasets too large to
materialize — the two capabilities this proposal is meant to exercise and
showcase end-to-end.
Why this env
list_splits/num_tasks/get_task/get_task_range, so the Task APIhas a concrete reference implementation beyond the importer templates.
dataset instead of downloading it, so TB-scale datasets (millions of rows,
hundreds of GB of images) can be served from a small Space.
vision-LLM policy. Fills a gap: most reference envs are text-only.
Proposed design
reset(split, index)→ observation with base64 image(target hidden) →
step(LatexOCRAction(latex=…))→ reward,done=True,ground truth revealed.
LATEX_OCR_MODE):materialize— split loaded + indexed;reset(split, index)random access.Best when the dataset fits on disk.
stream— sequential cursor over a streamed split, no full download;observations carry progress (
index / total / remaining / pct_done);num_taskscomes from dataset metadata only; no-repeat within a pass.reset(index=i)random access is intentionally unsupported (streaming issequential). This is the mode for very large / TB-scale datasets.
LatexOCRRubric):0.8·(1−CER) + 0.2·exact_match, whitespace-insensitive normalized character edit distance; pure-Python (no extra deps).
(image, latex)dataset works viaLATEX_OCR_DATASET(+ column env vars);default is
unsloth/LaTeX_OCR.Scope / non-goals
Environment+ the new Task API as-is.Validation
Verified end-to-end locally and on a deployed HF Space: Task API, streaming
cursor + live progress, no-repeat, and a real vision-LLM policy via the HF
Inference Router (mean reward ~0.77, incl. exact matches). Rubric has unit tests.