Skip to content

[AAI-237] Add correction score type#97

Open
kxzk wants to merge 3 commits into
mainfrom
feature/aai-237-add-correction-score-type
Open

[AAI-237] Add correction score type#97
kxzk wants to merge 3 commits into
mainfrom
feature/aai-237-add-correction-score-type

Conversation

@kxzk

@kxzk kxzk commented Jul 13, 2026

Copy link
Copy Markdown
Collaborator

TL;DR

Add TEXT and CORRECTION score data types to the existing flat score creation APIs, while keeping experiment evaluations limited to numeric/boolean/categorical.

Why

Langfuse corrections are replacement outputs stored as scores with dataType: "CORRECTION", name: "output", and a string value; the SDK previously supported only numeric, boolean, and categorical scores, so corrections (and text scores) couldn't be created from Ruby. Text scores are included in this slice so the general score type model matches the full Langfuse contract.

  • Types::SCORE_DATA_TYPES gains text: "TEXT" and correction: "CORRECTION"; a new explicit EXPERIMENT_SCORE_DATA_TYPES set keeps Langfuse::Evaluation from inferring its accepted types from the broader mapping.
  • ScoreClient requires a String for both types, enforces the documented 1–500-character limit for text (no invented limit for corrections), preserves the caller's score name, and emits TEXT/CORRECTION unchanged in the ingestion event. normalize_value was restructured into per-type helpers, dropping a rubocop complexity waiver.
  • Langfuse::Evaluation now rejects :text/:correction with a clear ArgumentError naming the experiment-accepted types (matching langfuse-python's intentional boundary).
  • Numeric/boolean/categorical normalization and all batching/sampling/flush behavior unchanged.

Live-validated against Langfuse Cloud: created a correction for a real observation and read it back via GET /api/public/v3/scores?dataType=CORRECTION&fields=subject,details (subject kind observation, name: "output", string value); TEXT score round-trips as well.

Checklist

  • Has label
  • Has linked issue
  • Tests added for new behavior
  • Docs updated (if user-facing)

Closes AAI-237


Note

Low Risk
Additive API and validation on score ingestion; experiment paths are explicitly narrowed but existing metric score behavior is unchanged.

Overview
Adds :text and :correction to the general score APIs (create_score, active trace/observation scoring) so Ruby can send Langfuse TEXT notes and CORRECTION replacement outputs, with docs in SCORING.md.

:text requires a 1–500 character string; :correction requires a string (no SDK length cap), a non-empty trace_id, and rejects session_id, dataset_run_id, and config_id. Ingestion events use dataType: "TEXT" / "CORRECTION"; the score name is left as the caller supplied it.

Value handling moves into Langfuse::ScoreValue (replacing inline normalization in ScoreClient). Types::SCORE_DATA_TYPES grows the two new types; EXPERIMENT_SCORE_DATA_TYPES keeps Langfuse::Evaluation limited to numeric/boolean/categorical, with explicit ArgumentError for :text and :correction (aligned with langfuse-python).

Reviewed by Cursor Bugbot for commit bb79dcf. Bugbot is set up for automated code reviews on this repo. Configure here.

Copilot AI review requested due to automatic review settings July 13, 2026 14:44
@kxzk kxzk added the enhancement New feature or request label Jul 13, 2026
@linear-code

linear-code Bot commented Jul 13, 2026

Copy link
Copy Markdown

AAI-237

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds support for Langfuse score dataType values TEXT and CORRECTION to the Ruby SDK’s flat score APIs, while explicitly keeping Langfuse::Evaluation (experiment metrics) limited to numeric/boolean/categorical types.

Changes:

  • Extend Types::SCORE_DATA_TYPES with :text and :correction, and introduce Types::EXPERIMENT_SCORE_DATA_TYPES for experiment-safe evaluation types.
  • Update ScoreClient value normalization/validation to handle text length limits (1–500) and correction string-only values, emitting TEXT/CORRECTION unchanged.
  • Update user-facing scoring docs and add specs covering the new types and the experiment-evaluation restriction.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
spec/langfuse/types_spec.rb Adds expectations for the expanded score type mapping and new experiment-only type set.
spec/langfuse/score_client_spec.rb Adds coverage for TEXT/CORRECTION validation and emitted ingestion payload fields.
spec/langfuse/evaluation_spec.rb Ensures Langfuse::Evaluation rejects :text/:correction with a clear error.
spec/langfuse/client_spec.rb Verifies Client#create_score forwards :text/:correction to ScoreClient unchanged.
lib/langfuse/types.rb Adds text/correction score types and defines EXPERIMENT_SCORE_DATA_TYPES.
lib/langfuse/score_client.rb Implements per-type normalization/validation for TEXT and CORRECTION.
lib/langfuse/observations.rb Updates scoring API docs to include the new score types.
lib/langfuse/evaluation.rb Switches validation to EXPERIMENT_SCORE_DATA_TYPES and clarifies behavior in docs/errors.
lib/langfuse/client.rb Updates client scoring API docs/examples to include TEXT and CORRECTION.
lib/langfuse.rb Updates module-level scoring API docs to include TEXT and CORRECTION.
docs/SCORING.md Documents TEXT and CORRECTION usage and the evaluation-type boundary.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread docs/SCORING.md Outdated
Comment on lines +117 to +118
with a diff view, and can be read back through the v3 scores API
(`data_type: "CORRECTION"`).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed — the read-back reference now shows the actual API query (GET /api/public/v3/scores?dataType=CORRECTION) instead of the Ruby-style keyword.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants