Skip to content

/performance failure rates divide months of never-rotated failures by days of rotated activity (81% shown, 0% real), and double-count failures in the denominator #2040

Description

@jacobo-ortiz

What breaks

/performance published an 81.2% failure rate for the Agent tool on this install. The true 7-day rate, measured from the same logs with a shared window, was 0% over 77 calls.

Root cause — two independent defects in LIFEOS/PULSE/Performance/module.ts

1. No date filter over two files with different retention. handleFailuresApi() reads both logs whole:

  • tool-failures.jsonl — never rotated (~4 months of rows on this install)
  • tool-activity.jsonl — rotated into OBSERVABILITY/archive/ (~9 live days)

So the numerator carries months of failures while the denominator carries days of successes. The published numbers reproduce exactly from that asymmetry (Agent 409 + 95 = 504 → 81.2%; WebFetch 819 + 683 = 1502 → 54.5%). handleSummaryApi()'s overallFailureRate has the same shape.

2. Failures are double-counted in the denominator. Line ~177 computes total = calls + fails with the comment "activity tracker may not count failures" — but failures ARE also written to tool-activity.jsonl (verified by causing one and finding the row in both files), so every failure inside the activity window is counted twice.

Suggested fix (applied locally, verified)

  • Filter both arrays to one cutoff (?days param, like handleCostApi already does; default 7 so the rotated archive fully covers the window).
  • Read archive/tool-activity-*.jsonl back alongside the live file so the denominator survives rotation.
  • Denominator Math.max(calls, fails) instead of calls + fails — correct when failures live in activity, and guards >100% rates for any log era where they didn't.
  • Same treatment for the daily trend and the summary's overallFailureRate.

After the fix, live values on this install: Agent 0% (77 calls), WebFetch 17.9% (324), overall 1.85% — matching an independent recomputation of the raw logs.

Why it matters

A dashboard tile reading "Agent 81% failure" invites exactly the wrong intervention — the number is an artifact of retention policy, not reliability. (The real composition of those historical failures, for what it's worth: ~72% are the harness's flat-roster rejection of teammates spawning teammates, ~24% the concurrency cap — instant rejections, not runtime failures.)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions