feat: add single-runtime prompt observations - #100
Draft
adityathebe wants to merge 4 commits into
Draft
Conversation
Add a versioned machine-oriented prompt observation contract that resolves and executes exactly one runtime while reporting execution failures as observation data. Capture provider-native reasoning effort at OpenAI and Codex dispatch boundaries, correlate brokered permission and tool lifecycle evidence, and preserve full disjoint usage, cost source, and raw timing without changing prompt run output. Amp-Thread-ID: https://ampcode.com/threads/T-01a0397c-1820-716d-9d2b-6223042c9b59
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Gavel summary
Totals: 0 passed · 0 failed · 0 skipped · - |
Gavel summary
Totals: 4392 passed · 0 failed · 12 skipped · 3m50s |
Genkit model middleware sees generic config before conversion, so a conversion failure could previously claim provider-native evidence without any provider call. Move OpenAI capture to the fully marshaled openai-go HTTP request, and report unknown/partial when that request cannot be inspected safely. Exercise both failed conversion with zero transport calls and successful native dispatch. Amp-Thread-ID: https://ampcode.com/threads/T-01a0397c-1820-716d-9d2b-6223042c9b59
Run both pinned OpenAI generator cases through Captain generic middleware. Any return to pre-conversion effort recording now makes the no-dispatch case or single-dispatch case fail. Amp-Thread-ID: https://ampcode.com/threads/T-01a0397c-1820-716d-9d2b-6223042c9b59
Buffered observation treated every ai.Response Usage value as present, so providers that omitted usage produced falsely known all-zero buckets. Carry native/event usage presence through the observation recorder, preserve nil versus known-zero usage in Genkit and Codex terminal events, and keep provider-reported cost independent of usage availability. Amp-Thread-ID: https://ampcode.com/threads/T-01a0397c-1820-716d-9d2b-6223042c9b59
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
captain prompt observe <prompt> --runtime <selector>contract for exactly one resolved runtimeopenai-go/chat/completionsrequest at the SDK transport boundary; failed Genkit conversion records no dispatch, while unsafe body inspection reportsunknownwith partial captureunknownwithout buckets while present all-zero usage remainsknownwith all five zero bucketscaptain prompt runbehaviorValidation
go test ./pkg/ai/provider/genkit -run 'TestOpenAI(ConversionFailureDoesNotRecordNativeDispatch|NativeDispatchRecordsMarshaledReasoningEffort)$' -v— passed; both cases run through Captain’s generic Genkit middleware; the conversion-failure case keepsreasoning_effort: highin generic config, proves zero transport calls and zero dispatch events, and reportsobserved: unknown; the success case proves the SDK HTTP doer receivedhighand reports evidence-backedobserved: knownpkg/cli,pkg/ai/provider/genkit, andpkg/ai/provider— passed; successful omitted usage staysunknownwith no buckets, while native/event-present zero usage remains non-nil and emits all five zero buckets through Genkit, generic stream coalescing, Codex CLI, and Codex app-server seamsgo vet— passedgolangci-lint run ./...— 0 issuesgo build ./cmd/captain— passedcaptain.observation/v1document with process exit 0, execution failure separated from three SDK dispatch attempts, and evidence-backedobserved: known/high; a no-effort run reported evidence-backedobserved: unsetgo test ./...passed except the unrelated order-dependentpkg/ai/provider/cmuxTestSessionStatsCacheColdCachesByMtimefailure (OutputTokens after rewrite = 20, want 100); the untouched test passes in isolationDeferred scope
unsupported/not_requestedstates with bounded empty events/artifacts. The MCP/Kubernetes event and artifact acceptance criteria are not complete in this PR.Addresses #99.
Companion: flanksource/gavel#84.