Skip to content

feat(review): optional delta summary comment on follow-up reviews - #459

Merged
devops-thiago merged 5 commits into
release/v0.6.0from
feat/327-followup-delta-summary
Aug 8, 2026
Merged

feat(review): optional delta summary comment on follow-up reviews#459
devops-thiago merged 5 commits into
release/v0.6.0from
feat/327-followup-delta-summary

Conversation

@devops-thiago

@devops-thiago devops-thiago commented Aug 8, 2026

Copy link
Copy Markdown
Owner

What type of PR is this?

  • 🐛 Bug fix
  • ✨ Feature
  • 📝 Documentation
  • 🔧 Refactor
  • 🚀 Performance
  • ✅ Test
  • 🔒 Security
  • 📦 Dependency update
  • 🏗️ CI/CD

Description

Follow-up reviews carry their signal in the review itself: only the first review posts a summary comment, so a maintainer on a busy PR has to read every inline thread to see what moved since the last bot pass. This adds an opt-in short delta comment on follow-up reviews.

FollowUpDeltaSummary.render(ReviewResult) is a pure function of the counts the follow-up pipeline already produced, and returns four lines:

## 🤖 ThrillhouseBot — changes since the last review

- **New findings this round:** 2
- **Previous findings resolved:** 1
- **Previous findings still open:** 3

plus the shared partial-coverage disclosure (ReviewResult.truncationDisclosure) when the diff was truncated. ReviewPublisher.publishFollowUpDelta(...) posts it, and ReviewOrchestrator calls it best-effort between the summary post and postReview.

Design decisions worth a reviewer's attention:

  • Skipped when nothing moved. "Delta" means the round raised a finding or closed one. Previous findings that merely stayed open are reported inside a comment that posts for one of those reasons, but never trigger one on their own — a pass that re-states the same open count is exactly the per-push noise this feature must not add.
  • Four gates before it posts: the flag is on, it is not a first review, no summary comment was posted this round (so /summary re-posts and superseded-round refreshes never get a delta beside them), and the delta is non-empty.
  • It never feeds summaryPosted. That flag gates the redundant-review skips in postReview and means "the PR already carries this round's verdict in a comment". A delta comment carries counts, not a verdict, so publishFollowUpDeltaBestEffort discards its return value; a failed summary post still leaves review posting enabled, exactly as before.
  • Its heading is deliberately not PrSummaryGenerator.SUMMARY_HEADING. That heading is the marker ReviewContextLoader.isBotSummaryComment uses to decide whether a review is the first one, and which comment the superseded path edits in place — a delta comment carrying it would be mistaken for a summary. A test pins that it is not.
  • Delta arithmetic. resolvedPreviousCount() is strictly the resolved status. justified is a maintainer's decline, not a fix, and superseded is an auto-close because the targeted code left the diff; counting either would overstate what the round fixed. Both are therefore in neither the resolved nor the still-open count.
  • Fail-soft. A failure posting the comment is swallowed and logged, like the summary post — it is enrichment, not the review, and must not abort before postReview.

Default is false; an untouched deployment keeps today's quiet follow-up review.

Known limitation — delta accuracy depends on #455

The still-open count this comment renders is ReviewResult.unresolvedPreviousCount(), and #455 documents that the previous-findings context is corrupted whenever a review round returns zero findings: FollowUpAnalyzer's fallback feeds the bot's own review body back as a "finding", which both drops real findings out of tracking (so they can never be counted resolved again) and inflates the unresolved count. This PR does not fix that — it is tracked separately — so while #455 is open the counts here inherit its inaccuracy in both directions.

Two things limit the blast radius, and both were checked rather than assumed:

  • A round whose only "movement" is a phantom carry-over renders nothing: the phantom lands as unresolved, so new findings and resolved are both zero and the no-delta guard fires. That is the exact shape of the fix(review): previous-findings fallback feeds the bot's own review body back as a "finding", losing real findings and inflating the unresolved count #455 repro (round N raises a finding, round N+1 raises none, round N+2 reports an inflated unresolved count), and noNewFindingsAndNothingResolvedRendersNothing covers it. The feature therefore never originates a comment off a miscounted carry-over; it can only display one alongside a genuine new-or-resolved delta.
  • Nothing here makes a wrong count harder to spot: the same number already drives the review body's unresolvedPreviousMessage and the APPROVE→COMMENT demotion.

The renderer's javadoc records the dependency so it is not lost.

Related Issues

Fixes #327

Depends for count accuracy on #455 (not fixed here).

How Has This Been Tested?

  • Unit tests
  • Integration tests
  • Manual testing

Full suite green (2060 tests), BugInstance size is 0, spotless clean. JaCoCo shows 100% line and branch coverage on every new method: render 8/8 lines and 4/4 branches, publishFollowUpDelta 10/10 lines and 8/8 branches, publishFollowUpDeltaBestEffort 9/9 lines including the catch, resolvedPreviousCount 1/1. cd website && npm ci && npm run build was reproduced for the README config-table row and reports "All internal links are valid."

Every new test was mutation-tested: production code was neutralized in a targeted way and the test had to go red. Test files were never edited to force a failure.

FollowUpDeltaSummaryTest

1. Widen the no-delta guard to newFindings == 0 && resolved == 0 && unresolvedPreviousCount() == 0, i.e. let a stalled round post:

FollowUpDeltaSummaryTest.noNewFindingsAndNothingResolvedRendersNothing:60
expected: <Optional.empty> but was: <Optional[## 🤖 ThrillhouseBot — changes since the last review

This is the anti-noise case, so it was mutated hardest. Previous findings that merely stayed open must not produce a comment on their own.

2. Replace the no-delta guard with if (false) — always render. Three red:

FollowUpDeltaSummaryTest.noNewFindingsAndNothingResolvedRendersNothing:60
FollowUpDeltaSummaryTest.emptyRoundWithNoPreviousStatusesRendersNothing:65
FollowUpDeltaSummaryTest.justifiedOrSupersededAloneIsNotADelta:74
    all: expected: <Optional.empty> but was: <Optional[## 🤖 ThrillhouseBot — changes since the last review

3. Make resolvedPreviousCount() also count justified and superseded:

FollowUpDeltaSummaryTest.justifiedOrSupersededAloneIsNotADelta:74
    expected: <Optional.empty> but was: <Optional[## 🤖 ThrillhouseBot — changes since the last review
FollowUpDeltaSummaryTest.newFindingsAloneRenderTheDelta:89
    (its "**Previous findings resolved:** 0" assertion)

4. Render the still-open line as a hardcoded 0:

FollowUpDeltaSummaryTest.resolvedPreviousFindingsAloneRenderTheDelta:105
FollowUpDeltaSummaryTest.newFindingsAloneRenderTheDelta:90
    (their "**Previous findings still open:** 1" assertions)

5. Drop ReviewResult.truncationDisclosure(...) from the body:

FollowUpDeltaSummaryTest.truncatedReviewDisclosesPartialCoverage:114

6. Always disclosetruncationDisclosure(omittedFiles() + 1). The negative case discriminates too:

FollowUpDeltaSummaryTest.untruncatedReviewCarriesNoDisclosure:122
FollowUpDeltaSummaryTest.truncatedReviewDisclosesPartialCoverage:114

7. DELTA_HEADING = PrSummaryGenerator.SUMMARY_HEADING:

FollowUpDeltaSummaryTest.deltaCommentIsNeverMistakenForTheSummaryComment:131
    ## 🤖 ThrillhouseBot PR Summary

ReviewPublisherTest — each clause of the four-part guard removed in turn

Drop !config.review().followUpSummary().enabled():

ReviewPublisherTest.followUpDeltaIsNotPostedWhenTheFeatureIsOff:163
    expected: <false> but was: <true>

Drop result.isFirstReview():

ReviewPublisherTest.followUpDeltaNeverDuplicatesTheFirstRunSummary:223
    expected: <false> but was: <true>

Drop summaryPosted:

ReviewPublisherTest.followUpDeltaIsSkippedWhenThisRoundAlreadyPostedASummary:225
    expected: <false> but was: <true>

The isFirstReview row is the one repaired in this PR. As first written the test passed isFirstReview = true and summaryPosted = true, so the short-circuit meant it stayed green with the first-review clause removed — it duplicated the already-posted-a-summary test instead of covering the acceptance criterion. It now drives the first-review path alone (summaryPosted = false, the shape a swallowed summary-post failure produces), asserts publishSummary posts while publishFollowUpDelta does not, and pins that the round leaves exactly one comment on the PR — the summary, verified by body.

ReviewOrchestratorTest — end-to-end through review()

Run with -Dtest='ReviewOrchestratorTest$ReviewErrorPaths'.

Never call publishFollowUpDeltaBestEffort from review():

shouldPostDeltaCommentOnFollowUpReviewWhenFollowUpSummaryEnabled
Wanted but not invoked:
commentClient.createComment(
    <any string>, <any string>, <any string>, <any string>, <any integer>,
    <custom argument matcher>
);
Actually, there were zero interactions with this mock.

Remove the try/catch around the delta post so the failure propagates — the review never posts and the run falls into failure handling:

shouldStillPostTheReviewWhenTheDeltaCommentFailsToPost
Wanted but not invoked:
reviewClient.createReview(
    <any string>, <any string>, <any string>, <any string>, <any integer>, <any>
);
However, there were exactly 2 interactions with this mock:
reviewClient.listReviews("Bearer test", "application/vnd.github+json", "owner", "repo", 42);
-> at ReviewContextLoader.fetchPriorReviews(...)

That second test is new in this PR: the catch in publishFollowUpDeltaBestEffort was the only uncovered branch in the new code. It also pins that the failed delta produces no follow-on failure-notice comment.

FollowUpSummaryDefaultOffTest

Set thrillhousebot.review.follow-up-summary.enabled=${REVIEW_FOLLOW_UP_SUMMARY_ENABLED:true} in application.properties:

FollowUpSummaryDefaultOffTest.followUpSummaryIsOffByDefault:37
    expected: <false> but was: <true>

Set @WithDefault("true") on FollowUpSummaryConfig.enabled() — survived, and that is correct. application.properties always supplies a value for this key, so the annotation default is unreachable in a running deployment. The test pins the effective default, which is the one that ships, and that is what the mutation above proves. This matches every other flag in ReviewConfig.

The full suite was run before each commit, not just the touched classes.

Checklist

  • My code follows the project's coding standards
  • I have performed a self-review of my own code
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have updated the documentation accordingly
  • My changes generate no new warnings or errors

Screenshots / Logs

Rendered comment on a truncated follow-up round:

## 🤖 ThrillhouseBot — changes since the last review

- **New findings this round:** 1
- **Previous findings resolved:** 0
- **Previous findings still open:** 0

> ⚠️ **Large PR — partial coverage.** 3 file(s) were omitted because the diff exceeded the size budget, so this covers only part of the diff.

Additional Notes

Acceptance criteria

  • ✅ Config flag default false@WithDefault("false") on ThrillhouseConfig.FollowUpSummaryConfig.enabled(), thrillhousebot.review.follow-up-summary.enabled=${REVIEW_FOLLOW_UP_SUMMARY_ENABLED:false} in application.properties, #REVIEW_FOLLOW_UP_SUMMARY_ENABLED=false in .env.example, and false in the README config table. All four agree, and FollowUpSummaryDefaultOffTest asserts through the resolved configuration rather than the annotation.
  • ✅ Delta summary posted on follow-up when enabled and delta non-empty — followUpDeltaIsPostedWhenEnabledAndTheDeltaIsNonEmpty at the publisher, shouldPostDeltaCommentOnFollowUpReviewWhenFollowUpSummaryEnabled end-to-end.
  • ✅ Does not duplicate the first-run summary comment — the first-review clause is now covered on its own path, and the test pins that a first review leaves exactly one comment (the summary). The heading test additionally pins that the delta can never be mistaken for a summary by isBotSummaryComment, which is what would otherwise corrupt first-review detection on the next round.
  • ✅ Tests for enabled/disabled and delta detection — off / on / zero-delta / first-review / summary-already-posted at the publisher, plus seven pure-render cases and two orchestrator paths.

Scope

REVIEW_FOLLOW_UP_SUMMARY_ENABLED is documented in the README config table and .env.example. website/src/content/docs/configuration.md mirrors that README section through its include marker, so no separate docs edit was needed, and the site build was run to confirm. No CHANGELOG.md entry, matching the other v0.6.0 wave PRs (#450, #451, #453).

Worth knowing

  • Each qualifying follow-up round posts a new comment rather than editing one in place, so a long-running PR with many pushes accumulates delta comments. That is the literal reading of the issue and the reason the feature is opt-in and skips zero-delta rounds; an edit-in-place variant would need its own marker and is a candidate follow-up if the accumulation proves noisy.
  • A round whose only movement is a maintainer declining a finding (justified) posts nothing, by the arithmetic above. The three counts therefore do not necessarily sum to the previous-finding total — each label states exactly what it counts, so nothing is misreported, but a reviewer should be aware of it.
  • On a superseded round the summary is refreshed in place and the delta is skipped. If that refresh throws, publishSummaryBestEffort swallows it and reports false, so a delta comment can post on that round instead. That is deliberate: enrichment degrading to a shorter form beats degrading to nothing.

Only the first review posts a summary, so a maintainer on a busy PR cannot
see what changed since the last bot pass without reading every inline thread.
An opt-in follow-up delta comment reports new, resolved and still-open finding
counts, disabled by default and skipped entirely when the delta is empty.

Refs #327
The first-review test for the delta comment passed both `isFirstReview`
and `summaryPosted` as true, so the short-circuit meant it stayed green
with the first-review check removed from the guard — it proved only that
a round which already posted a summary skips the delta, which a separate
test already covers. It now drives the first-review path on its own
(`summaryPosted == false`, the shape a swallowed summary-post failure
produces) and pins that the round leaves exactly one comment on the PR:
the summary itself.

The orchestrator delta test likewise verified only that a matching
comment was created, which an extra non-matching summary comment would
not have contradicted; it now also pins the total comment count. Adds
the missing fail-soft case: a delta comment whose post throws must be
swallowed so the review still reaches the PR and the session completes
rather than failing, which was the one uncovered branch in the new code.

Also corrects the superseded wording in the renderer's javadoc — the
caller skips the delta when the summary re-post lands, not merely
because a finding was superseded — and records that the counts inherit
the previous-findings defect tracked in #455.

Refs #327
@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

@thrillhousebot

thrillhousebot Bot commented Aug 8, 2026

Copy link
Copy Markdown

🤖 ThrillhouseBot PR Summary

What this PR does

Adds an opt-in delta summary comment on follow-up reviews, displaying new, resolved, and still-open finding counts, with a truncation disclosure when the diff was partial. Controlled by a new configuration flag, disabled by default to preserve existing quiet follow-up behavior.

Control-Flow Diagram

🔀 Show diagram
flowchart TD
  A["ReviewOrchestrator.review()"] --> B["publishSummaryBestEffort"]
  B --> C["summaryPosted"]
  A --> D["publishFollowUpDeltaBestEffort(auth, req, result, summaryPosted)"]
  D --> E{"enabled && !firstReview && !summaryPosted?"}
  E -- no --> F["return"]
  E -- yes --> G["FollowUpDeltaSummary.render(result)"]
  G --> H{"delta present?"}
  H -- no --> I["log no delta, return"]
  H -- yes --> J["commentClient.createComment(...)"]
  J --> K["return true"]
Loading

Changes Overview

  • Files changed: 12
  • Lines added: +662
  • Lines removed: -5

Changed Files

File Change Summary
.env.example Modified Documented new REVIEW_FOLLOW_UP_SUMMARY_ENABLED env var.
README.md Modified Added config table row for the new flag.
src/main/java/dev/thiagogonzaga/thrillhousebot/config/ThrillhouseConfig.java Modified Added FollowUpSummaryConfig interface with default false.
src/main/java/dev/thiagogonzaga/thrillhousebot/review/FollowUpDeltaSummary.java Added New utility class that renders the delta comment body from ReviewResult counts, returning empty when nothing moved.
src/main/java/dev/thiagogonzaga/thrillhousebot/review/ReviewOrchestrator.java Modified Calls new best-effort delta comment post after summary publishing, swallowing failures.
src/main/java/dev/thiagogonzaga/thrillhousebot/review/ReviewPublisher.java Modified Added publishFollowUpDelta method with guards (feature flag, first-review, summary-already-posted, delta detection).
src/main/java/dev/thiagogonzaga/thrillhousebot/review/ReviewResult.java Modified Added resolvedPreviousCount() method counting 'resolved' statuses.
src/main/resources/application.properties Modified New property thrillhousebot.review.follow-up-summary.enabled defaulting to false.
src/test/java/dev/thiagogonzaga/thrillhousebot/config/FollowUpSummaryDefaultOffTest.java Added Integration test asserting the feature is off in the default Quarkus profile.
src/test/java/dev/thiagogonzaga/thrillhousebot/review/FollowUpDeltaSummaryTest.java Added Unit tests for delta detection, rendering, truncation disclosure, and heading separation.
src/test/java/dev/thiagogonzaga/thrillhousebot/review/ReviewOrchestratorTest.java Modified Two integration tests covering the delta comment happy path and failure handling (delta post fails but review still completes).
src/test/java/dev/thiagogonzaga/thrillhousebot/review/ReviewPublisherTest.java Modified Added tests for publishFollowUpDelta posting conditions (feature off, enabled/delta, skip on no delta, first-review, summary-already-posted).

Risk Assessment

Risk Count
🔴 Critical 0
🟠 High 0
🟡 Medium 0
🔵 Low 0

No new issues found in this PR, but the review cannot be approved until CI is confirmed green.

⚠️ CI Checks Status

Some checks are still pending or have failed:

Check Type Status Detail
test check-run ⏳ Pending -
frontend check-run ⏳ Pending -
actionlint check-run ⏳ Pending -
format check-run ⏳ Pending -
trivy check-run ⏳ Pending -
changes check-run ⏳ Pending -
build check-run ⏳ Pending -
dependency-review check-run ⏳ Pending -

Automated review by ThrillhouseBot. Reply with /review to re-run.

@thrillhousebot thrillhousebot Bot added enhancement New feature or request java Pull requests that update java code testing Test coverage and test quality labels Aug 8, 2026
@codecov

codecov Bot commented Aug 8, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

…llowup-delta-summary

# Conflicts:
#	README.md

@thrillhousebot thrillhousebot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything's coming up Thrillhouse! 🎉

No issues found in this PR.

…nent

#460 added a pathInstructions component to the RepoSettings record after
PrImprovementServiceTest was written, so its two-argument construction no
longer compiles and testCompile fails on release/v0.6.0 and every branch
merging it.

Both call sites exercise the per-repo ignore filter and declare no scoped
review rules, so they pass an empty path-instructions list and keep asserting
exactly what they asserted before.

Refs #327

@thrillhousebot thrillhousebot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything's coming up Thrillhouse! 🎉

No issues found in this PR.

@sonarqubecloud

sonarqubecloud Bot commented Aug 8, 2026

Copy link
Copy Markdown

@devops-thiago
devops-thiago merged commit 94499bb into release/v0.6.0 Aug 8, 2026
23 of 26 checks passed
@devops-thiago
devops-thiago deleted the feat/327-followup-delta-summary branch August 8, 2026 20:42
devops-thiago added a commit that referenced this pull request Aug 8, 2026
…ared-command-batching

Resolves the config table in README.md, where this branch's rewrite of the
REVIEW_MAX_INPUT_TOKENS row (it now covers /describe and /changelog too) and
the REVIEW_FOLLOW_UP_SUMMARY_ENABLED row #459 added landed on the same lines.
Both rows are kept.

No source conflict: #464 had already restored the three-argument RepoSettings
construction on base, and this branch carried the identical change, so the
ignore-filter tests of /describe, /changelog and /improve merged as-is. No
batching behaviour changes.
devops-thiago added a commit that referenced this pull request Aug 9, 2026
…463)

## What type of PR is this?

- [ ] 🐛 Bug fix
- [x] ✨ Feature
- [x] 📝 Documentation
- [x] 🔧 Refactor
- [ ] 🚀 Performance
- [ ] ✅ Test
- [ ] 🔒 Security
- [ ] 📦 Dependency update
- [ ] 🏗️ CI/CD

## Description

`max-diff-lines` predates token budgeting and had become a second,
cruder ceiling sitting in front of it. The review path stopped using it
when map-reduce (#53) landed, and `/improve` was moved onto that in #316
— but `/describe` and `/changelog` still shrank a large PR to the first
`max-diff-lines` of its rendered diff. A description was therefore
written from a partial diff, and a CHANGELOG entry drafted from one,
with whole files never reaching the model at all.

**The seam.** The three methods `/improve` proved — batch planning, the
shared prompt overhead, and the per-repo ignore re-filter — are lifted
into `AbstractPrSuggestionGenerator`, parameterised by each command's
own prompt constants. `/improve` is refactored onto the lifted versions
rather than keeping a private copy. Every on-request suggestion command
now plans batches over the reviewable **file list** under the per-call
token budget and makes one call per batch. One call:

```java
var plan = planBatches(reviewable, inputs, ownSystemPrompt, ownUserPrompt, reservedCalls);
```

**The reduce step is per-command, because the reductions genuinely
differ.** Batching is only the map step:

| Command | Reduce | Extra AI call? |
|---|---|---|
| `/describe` | Per-batch partial descriptions **synthesized** into one
coherent title + description | Yes — reserved, spent only when >1 batch
|
| `/changelog` | Per-batch candidate entries **merged** into one entry |
Yes — reserved, spent only when >1 candidate |
| `/improve` | Local union of per-batch suggestions, deduped by
`file:line` | No |
| `/generate-tests` | Local union of per-batch test files, deduped by
path | No |

Nothing is concatenated. Stapling `/describe`'s partials together
repeats the overview once per part and reads as several pull requests;
for `/changelog`, a deterministic merge could unify headings and drop
identical bullets, but the duplicates that actually arise are *not*
identical — two batches that saw different files of one feature describe
that change in two different sentences, which only a reader that
understands them can collapse. Both reduce calls are reserved out of
`max-ai-calls` up front, the same way the review path reserves one for
its summary, so a run never exceeds the ceiling of one review; a
single-batch PR still costs exactly one call.

**Also in this change**

- The shared overhead is assembled from each command's **own** prompts.
Sizing a batch against another command's prompts would let an
"in-budget" batch overshoot the real input limit.
- Coverage disclosure now comes from `BudgetPlan.omittedFiles()` /
`clippedFiles()` — files **named**, not counted — keeping #296's
wording. `Inputs.omittedFiles` (the line-cap count) is gone, so nothing
can reach for the wrong number.
- Coverage on a huge PR is bounded by `max-ai-calls`, not the file list.
Files that never got a batch are named. When *no* file fit any batch at
all, every command says so and names the files rather than going silent
— a misconfigured budget must not look like a bot that ignored the
command. An empty plan that omitted *nothing*, because the repository
ignores every changed file, is the opposite case and posts nothing.
- Per-repo ignore patterns (#449) are applied on top of the global set
for every command, and the filtered list stays authoritative for
everything downstream — batches *and* the line map alike. That is the
bug #452's audit found; the lifted method carries the invariant in its
javadoc.
- `max-input-tokens <= 0` keeps budgeting off as a single uncapped batch
rather than regressing to the line-capped string.
- A batch whose model call fails is skipped rather than failing the run,
and the shortfall is disclosed.
- Sizing callers reach the prompts through `systemPrompt()` /
`userPrompt()` accessors: a reference to a `static final String` is
inlined into the caller's class file at compile time, and a third copy
of a multi-kilobyte prompt trips SpotBugs'
`HSC_HUGE_SHARED_STRING_CONSTANT`. The accessors are deliberately *not*
named `system()` / `user()` — differing from the constant only by
capitalization reads as a typo at the call site.

**`/add-docs` is deliberately out of scope.** It does not extend
`AbstractPrSuggestionGenerator`; it orders its loading around a hard
head-SHA precondition (every output is an inline suggestion, so no head
SHA means nothing postable and a distinct user-facing message), and it
feeds its assistant a different input set — project stack, a combined
`PromptSections.prContext(...)` block, and a pre-rendered instructions
section built from `ResolvedInstructions` rather than the content string
the shared `Inputs` carries. Folding it in therefore means changing the
shared `Inputs` contract at the same time as first lifting the seam, on
the command that posts committable edits. It is worth doing and should
be tracked separately; `/add-docs` remains line-capped and the README
now says so precisely. Note it also still has **no per-repo ignore
filter at all**, which is worth carrying into that follow-up.

**Path-scoped instructions (#460) are not part of what a command batch
carries.** `PathScopedInstructions` is resolved only by
`ReviewContextLoader` and rendered only into `ReviewPromptAssembler`'s
trailing-guidance slot, so it reaches the review prompt and nothing
else. The `repoInstructions` slot of the batched commands is fed solely
by `InstructionsResolver.resolve(...).content()` — the global
instructions file — which `sharedPromptOverhead(...)` already counts in
full. No batch is mis-sized by the scoped rules.

## `/generate-tests` is migrated onto the seam in this PR

#461 merged before this one, so `/generate-tests` landed on
`release/v0.6.0` still line-capped and still reading
`Inputs.omittedFiles`, which this PR removes. The migration #461's agent
was going to perform *after* this merged is therefore done **here** —
there is no "later", because without it base does not compile.
`UnitTestGenerator` now:

1. resolves its effective file list once via
`respectPerRepoIgnores(...)` and plans from it,
2. plans token-budgeted batches and sends `batch.text()` per batch,
3. discloses coverage from `disclosure(plan)`,
4. reserves **0** calls — its reduce is a local union, so the whole
`max-ai-calls` allowance buys batches.

**Two things this surfaced that are worth reading closely.**

**The shared overhead was not sufficient for this command, and using it
unchanged would have been a real bug.** `sharedPromptOverhead(...)`
counts system + user + fence + title + body + instructions.
`/generate-tests` also sends the resolved **project stack** on every
call — dependency manifests, kilobytes, not a rounding error — so the
estimate would have undercounted every batch by the size of the stack
and let "in-budget" batches overshoot the model's real input limit. That
is precisely the failure the overhead exists to prevent.
`planBatches(...)` therefore gains a six-argument form taking the
command's own extra per-call sections, and `/generate-tests` declares
the stack there. A future adopter with its own extra section must do the
same rather than reach for the five-argument form.

**It uses its own prompt templates, not the shared ones.**
`UnitTestAssistant` is annotated with `UnitTestAssistantPrompts.SYSTEM`
/ `UnitTestAssistantPrompts.USER` — it does **not** share
`PrSuggestionPrompts.USER`, because its user template carries the
project-stack section. Sizing its batches against the shared user
template would measure the wrong prompt. `UnitTestAssistantPrompts` had
no accessors (it is new from #461), so `systemPrompt()` / `userPrompt()`
are added to it — SpotBugs failed the build without them, exactly as the
accessor javadoc predicts.

**Why dedupe by path rather than merge.** Batches partition the file
list, so two batches usually propose disjoint test paths. When they do
collide, each proposal's `code` is a *complete* compilable file —
package, imports and fixtures included, posted verbatim to paste at that
path — so two of them at one path are alternatives, not additions.
Rendering both would invite pasting the second over the first and
silently losing the first's cases, and merging them properly would need
a model call for a rare collision. The first wins and the rest are
counted in a disclosure line, so the maintainer can re-run for the
others.

## Related Issues

Fixes #457

## How Has This Been Tested?

- [x] Unit tests
- [ ] Integration tests
- [ ] Manual testing

Every new behavior was validated red/green: the test was written, the
production change was mutated to neutralize exactly that behavior, the
test was confirmed to **fail**, and the mutation was reverted to confirm
it passes.

**One mutation initially stayed green and the test was rewritten.**
`proposesTestsForFilesThatTheLineCapWouldHaveDroppedEntirely` first
asserted only that each batch *contained* its file — which is also true
when every call is handed the whole-PR diff, the very behavior being
replaced. Strengthened to assert the partition (batch 1 contains `Foo`
and **not** `Other`, batch 2 the reverse), it goes red properly:

```
[ERROR] UnitTestGeneratorTest.proposesTestsForFilesThatTheLineCapWouldHaveDroppedEntirely:439 [[THRILLHOUSEBOT-UNTRUSTED-DATA-3d24804df8d0a8a72a18bb0d9f6a121f]]
```

**The project stack is counted in the budget.** Mutation: use the
five-argument `planBatches(...)`, leaving the stack out of the overhead.
With a 20k-character stack no file can honestly fit, so the correct run
makes no call at all; the mutant ships batches that overshoot:

```
[ERROR] UnitTestGeneratorTest.countsTheProjectStackInTheBudgetSoBatchesAreNotOversized:496
No interactions wanted here:
```

**Per-repo ignores stay authoritative.** Mutation: plan from
`inputs.reviewableFiles()` instead of the filtered list.

```
[ERROR] UnitTestGeneratorTest.leavesFilesTheRepositoryAskedTheBotToIgnoreOutOfScope:520 [[THRILLHOUSEBOT-UNTRUSTED-DATA-5c207a4fc3e69a526faf47a8f7df5769]]
[ERROR] UnitTestGeneratorTest.staysSilentWhenEveryChangedFileIsOutOfScope:574
```

**Same-path proposals are deduped.** Mutation: drop the `seenPaths`
guard.

```
[ERROR] UnitTestGeneratorTest.keepsOneProposalPerPathAndSaysHowManyWereLeftOut:479 ## 🤖 ThrillhouseBot — suggested unit tests
```

**Disclosure comes from the budget plan.** Mutation: `disclosure(plan)`
returns `""` and the empty-plan branch returns `null`.

```
[ERROR] UnitTestGeneratorTest.disclosesPartialCoverageEvenWhenNoTestsWereProposed:321 🧪 ThrillhouseBot found nothing in this PR's changes that warrants a new unit test. ==> expected: <true> but was: <false>
[ERROR] UnitTestGeneratorTest.namesTheFilesLeftUncoveredWhenTheBatchBudgetRunsOut:303 ## 🤖 ThrillhouseBot — suggested unit tests
[ERROR] UnitTestGeneratorTest.namesTheFilesWhenTheBudgetCouldNotCoverASingleOne:555 expected: not <null>
```

Earlier rounds for `/describe`, `/changelog` and `/improve` (batch text
vs. line-capped render, synthesis vs. concatenation, reserved reduce
call, per-repo ignores, disclosure, budgeting-disabled, nothing-covered,
merge declines) all went red as recorded before; the `<= 0` guard
mutation remains the one that does not, because `max-input-tokens=0`
reaches `Integer.MAX_VALUE` down the fall-through path and yields the
same single batch.

Build results, on the merge of `release/v0.6.0` at `ad36d22` (#458,
#460, #464, #459, #461):

```
./mvnw -B spotless:apply                                   # clean
./mvnw -B clean compile spotbugs:check spotless:check      # BUILD SUCCESS
./mvnw -B clean test                                       # Tests run: 2226, Failures: 0, Errors: 0, Skipped: 0
cd website && npm run build                                # "All internal links are valid."
```

## Checklist

- [x] My code follows the project's coding standards
- [x] I have performed a self-review of my own code
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the documentation accordingly
- [x] My changes generate no new warnings or errors

## Notes for anything still in flight

`Inputs.omittedFiles` is gone. It carried how many files the
`max-diff-lines` render dropped; once a command plans its own
token-budgeted batches that number describes a render nothing sends to a
model, so it is not merely redundant but wrong. Nothing on
`release/v0.6.0` or in this tree still reads it.

Any command extending `AbstractPrSuggestionGenerator` that is still in
flight needs the same four steps `/generate-tests` just took:

1. `disclosure(plan)` rather than a line-cap count.
2. `planBatches(reviewable, inputs, <its own system prompt>, <its own
user prompt>, reservedCalls)` — its **own** prompt constants, and the
six-argument form if it repeats a section the shared overhead does not
know about. Check the merged tree for whether the class exposes
`systemPrompt()` / `userPrompt()` accessors or only constants; adding
them is required if a sizing reference would inline a third copy.
3. `batch.text()` per batch, never `inputs.diff()`.
4. `respectPerRepoIgnores(target, COMMAND, inputs.reviewableFiles())`,
with that same list used for anything that anchors onto the diff.

## Additional Notes

**Operator-visible cost change.** `/describe` and `/changelog` on a PR
that needs more than one batch now cost one more model call than the
batches alone, reserved out of `REVIEW_MAX_AI_CALLS`, so the ceiling per
run is unchanged. `/improve` and `/generate-tests` reserve nothing.
Documented in the config table, the "AI call budget" section, the
command prose, and the Known limitations bullet.

**The ignore filter is authoritative for the line resolver, not just the
planner.** `/improve` threads the resolved list into both
`planBatches(...)` and `post(...)`, where the resolver is built as `new
DiffLineResolver(diffFormatter().patchesByReviewableFiles(reviewable))`
— never from `inputs.reviewableFiles()`.
`PrImprovementServiceTest.neverCommitsASuggestionToAFileTheRepositoryAskedTheBotToIgnore`
pins it. `/describe`, `/changelog` and `/generate-tests` build no line
map (a proposed test file is a new file with no diff line to anchor to),
so they cannot exercise it, but the rule is stated in
`respectPerRepoIgnores(...)`'s javadoc for future adopters.

**New prompts.**
`PrDescribeAssistantPrompts.SYNTHESIS_SYSTEM`/`SYNTHESIS_USER` and
`ChangelogAssistantPrompts.MERGE_SYSTEM`/`MERGE_USER`, with matching
`synthesize(...)` / `merge(...)` methods on the assistants. Both user
templates mirror `PrSuggestionPrompts.USER` — same context sections,
same random-fence untrusted-data block — with the partials/candidates in
place of the diff. `AiServicePromptRenderingTest` drives both through
the real rendering pipeline and asserts every `@V` reaches the message.

**No new config keys.**
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request java Pull requests that update java code testing Test coverage and test quality

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant