[AAI-237] Add correction score type#97
Open
kxzk wants to merge 3 commits into
Open
Conversation
There was a problem hiding this comment.
Pull request overview
Adds support for Langfuse score dataType values TEXT and CORRECTION to the Ruby SDK’s flat score APIs, while explicitly keeping Langfuse::Evaluation (experiment metrics) limited to numeric/boolean/categorical types.
Changes:
- Extend
Types::SCORE_DATA_TYPESwith:textand:correction, and introduceTypes::EXPERIMENT_SCORE_DATA_TYPESfor experiment-safe evaluation types. - Update
ScoreClientvalue normalization/validation to handle text length limits (1–500) and correction string-only values, emittingTEXT/CORRECTIONunchanged. - Update user-facing scoring docs and add specs covering the new types and the experiment-evaluation restriction.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| spec/langfuse/types_spec.rb | Adds expectations for the expanded score type mapping and new experiment-only type set. |
| spec/langfuse/score_client_spec.rb | Adds coverage for TEXT/CORRECTION validation and emitted ingestion payload fields. |
| spec/langfuse/evaluation_spec.rb | Ensures Langfuse::Evaluation rejects :text/:correction with a clear error. |
| spec/langfuse/client_spec.rb | Verifies Client#create_score forwards :text/:correction to ScoreClient unchanged. |
| lib/langfuse/types.rb | Adds text/correction score types and defines EXPERIMENT_SCORE_DATA_TYPES. |
| lib/langfuse/score_client.rb | Implements per-type normalization/validation for TEXT and CORRECTION. |
| lib/langfuse/observations.rb | Updates scoring API docs to include the new score types. |
| lib/langfuse/evaluation.rb | Switches validation to EXPERIMENT_SCORE_DATA_TYPES and clarifies behavior in docs/errors. |
| lib/langfuse/client.rb | Updates client scoring API docs/examples to include TEXT and CORRECTION. |
| lib/langfuse.rb | Updates module-level scoring API docs to include TEXT and CORRECTION. |
| docs/SCORING.md | Documents TEXT and CORRECTION usage and the evaluation-type boundary. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+117
to
+118
| with a diff view, and can be read back through the v3 scores API | ||
| (`data_type: "CORRECTION"`). |
Collaborator
Author
There was a problem hiding this comment.
Fixed — the read-back reference now shows the actual API query (GET /api/public/v3/scores?dataType=CORRECTION) instead of the Ruby-style keyword.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DRAdd
TEXTandCORRECTIONscore data types to the existing flat score creation APIs, while keeping experiment evaluations limited to numeric/boolean/categorical.WhyLangfuse corrections are replacement outputs stored as scores with
dataType: "CORRECTION",name: "output", and a string value; the SDK previously supported only numeric, boolean, and categorical scores, so corrections (and text scores) couldn't be created from Ruby. Text scores are included in this slice so the general score type model matches the full Langfuse contract.Types::SCORE_DATA_TYPESgainstext: "TEXT"andcorrection: "CORRECTION"; a new explicitEXPERIMENT_SCORE_DATA_TYPESset keepsLangfuse::Evaluationfrom inferring its accepted types from the broader mapping.ScoreClientrequires aStringfor both types, enforces the documented 1–500-character limit for text (no invented limit for corrections), preserves the caller's score name, and emitsTEXT/CORRECTIONunchanged in the ingestion event.normalize_valuewas restructured into per-type helpers, dropping a rubocop complexity waiver.Langfuse::Evaluationnow rejects:text/:correctionwith a clearArgumentErrornaming the experiment-accepted types (matching langfuse-python's intentional boundary).Live-validated against Langfuse Cloud: created a correction for a real observation and read it back via
GET /api/public/v3/scores?dataType=CORRECTION&fields=subject,details(subject kindobservation,name: "output", string value); TEXT score round-trips as well.ChecklistCloses AAI-237
Note
Low Risk
Additive API and validation on score ingestion; experiment paths are explicitly narrowed but existing metric score behavior is unchanged.
Overview
Adds
:textand:correctionto the general score APIs (create_score, active trace/observation scoring) so Ruby can send Langfuse TEXT notes and CORRECTION replacement outputs, with docs inSCORING.md.:textrequires a 1–500 character string;:correctionrequires a string (no SDK length cap), a non-emptytrace_id, and rejectssession_id,dataset_run_id, andconfig_id. Ingestion events usedataType: "TEXT"/"CORRECTION"; the score name is left as the caller supplied it.Value handling moves into
Langfuse::ScoreValue(replacing inline normalization inScoreClient).Types::SCORE_DATA_TYPESgrows the two new types;EXPERIMENT_SCORE_DATA_TYPESkeepsLangfuse::Evaluationlimited to numeric/boolean/categorical, with explicitArgumentErrorfor:textand:correction(aligned with langfuse-python).Reviewed by Cursor Bugbot for commit bb79dcf. Bugbot is set up for automated code reviews on this repo. Configure here.