Skip to content

[Env Request] LaTeX OCR — multimodal environment on the new Task API + dataset streaming #1002

Description

@adithya-s-k

Summary

Add a multimodal (vision + text) dataset-backed environment: the agent is
shown an image of a mathematical expression and must return its LaTeX
source. Reward is computed server-side against the hidden ground-truth LaTeX.

This environment is built specifically on top of the newly introduced Task
API
(#726) and adds dataset streaming for datasets too large to
materialize — the two capabilities this proposal is meant to exercise and
showcase end-to-end.

Why this env

  1. Exercises the new Task API — a real, dataset-backed consumer of
    list_splits / num_tasks / get_task / get_task_range, so the Task API
    has a concrete reference implementation beyond the importer templates.
  2. Dataset streaming — a first environment that streams a Hugging Face
    dataset instead of downloading it, so TB-scale datasets (millions of rows,
    hundreds of GB of images) can be served from a small Space.
  3. Multimodal — image input + text (LaTeX) output, gradeable with any
    vision-LLM policy. Fills a gap: most reference envs are text-only.

Proposed design

  • Episode (bandit): reset(split, index) → observation with base64 image
    (target hidden) → step(LatexOCRAction(latex=…)) → reward, done=True,
    ground truth revealed.
  • Task API modes (LATEX_OCR_MODE):
    • materialize — split loaded + indexed; reset(split, index) random access.
      Best when the dataset fits on disk.
    • streamsequential cursor over a streamed split, no full download;
      observations carry progress (index / total / remaining / pct_done);
      num_tasks comes from dataset metadata only; no-repeat within a pass.
      reset(index=i) random access is intentionally unsupported (streaming is
      sequential). This is the mode for very large / TB-scale datasets.
  • Reward (LatexOCRRubric): 0.8·(1−CER) + 0.2·exact_match, whitespace-
    insensitive normalized character edit distance; pure-Python (no extra deps).
  • Custom Gradio "Try it" tab: get a task image, type LaTeX, see the score.
  • Any (image, latex) dataset works via LATEX_OCR_DATASET (+ column env vars);
    default is unsloth/LaTeX_OCR.

Scope / non-goals

  • No new core APIs — uses the existing Environment + the new Task API as-is.
  • Semantic/rendered reward (render-and-compare) is out of scope for v1.

Validation

Verified end-to-end locally and on a deployed HF Space: Task API, streaming
cursor + live progress, no-repeat, and a real vision-LLM policy via the HF
Inference Router (mean reward ~0.77, incl. exact matches). Rubric has unit tests.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions