Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,24 @@ All notable changes to ThrillhouseBot.

## [Unreleased]

## [0.6.3] β€” 2026-08-15

Follow-ups from the round-7 dogfood corpus, on two surfaces: the GitHub write
path, and what a review says about its own work. A comment GitHub throttles is
now outlasted and its refusal explained, a finding whose inline comment cannot
land keeps a working review thread, and a review can no longer claim coverage
or closures its audit never made. The docs site also deploys itself on
release. No configuration changes; upgrading is a redeploy.

### Fixed

- **A comment GitHub refuses is outlasted, and the refusal reason reaches the log** (#722): the content-creation secondary limit was measured blocking for 72 seconds while the write retry's budget spanned 60, and both ways of deriving a delay without a `Retry-After` undershot it β€” the linear fallback summed to 30 seconds, and a stale `x-ratelimit-reset` yielded zero, spending every attempt while GitHub was still refusing. The retry now makes four attempts and floors a derived delay at 30 seconds during a content-creation block; an explicit `Retry-After` still wins outright, and a genuinely exhausted primary window keeps its own reset instant. Every rejection now logs GitHub's status and message at warning level, so a throttle is distinguishable from a rejected position β€” the round that motivated this left 29 rejections undiagnosable and blamed line anchoring for them. When the with-suggestion and without-suggestion attempts fail differently, both reasons are kept
- **A finding whose inline comment cannot land posts a file-level thread, not a bare bullet** (#712): the bullet carried no code context, no suggestion block and no review thread, and everything thread-dependent died with it β€” the finding could not be declined, a status note had nowhere to land, and clearing it acknowledged a count rather than a name. The suspected anchoring defect was ruled out: the same file and line posted successfully at the same commit, which is what pointed to the throttle above
- **An overturned decline is disclosed on every review body, and a round names the findings it closed** (#713, #714): the note explaining that a decline was overturned on diff evidence rode only the no-new-findings body, so a round that also raised a finding dropped it, and a maintainer who wrote a considered decline could not tell whether it was read, rebutted, or missed. The note now rides the with-findings bodies too, above the partial-coverage banner. The resolved tally likewise names each finding it closed instead of reporting a bare count
- **Verification coverage counts only the verdicts the audit acted on** (#710): the coverage record asked whether some verdict carried a finding's id, but `apply()` acts only on `confirmed`, `downgraded` and `rejected` β€” a blank or unrecognized verdict falls open and the finding posts unscreened. Such findings were counted as screened anyway, so a review could claim full verification over a set it never ruled on. One decision normalizer now feeds both the switch and the count, and an unrecognized verdict counts as unverified, which errs toward an over-cautious clause rather than silence
- **The verifier sees the PR description it judges description-gap findings against** (#711, first part): a finding weighing the author's stated intent against the code had half its claim in material the verifier never received, and the verifier's own prompt tells it to reject a claim whose material is missing β€” so survival came down to whether the model noticed the absence. Five planted description-versus-code mismatches split three kept, two rejected, on identical grounds. The verifier now receives the PR title and description under the same untrusted-data fencing the reviewer uses
- **The docs site deploys itself on release** (#717): the `release: published` trigger in `docs.yml` has fired zero times in the repository's history, because the release is created with the workflow's own `GITHUB_TOKEN` and GitHub starts no workflows for events that token raises β€” every site deploy through v0.6.2 was manual. `release.yml` now dispatches the docs build against the release tag once the release exists, gated on `update_latest` so a patch cut on an older line does not republish the site; the job fails loudly when the dispatch is refused

## [0.6.2] β€” 2026-08-14

Follow-ups to the review threads on 0.6.1, plus the first piece of the release
Expand Down
2 changes: 1 addition & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

<groupId>dev.thiagogonzaga.thrillhousebot</groupId>
<artifactId>thrillhousebot</artifactId>
<version>0.6.3-SNAPSHOT</version>
<version>0.6.3</version>

<properties>
<compiler-plugin.version>3.15.0</compiler-plugin.version>
Expand Down
Binary file added website/src/assets/0.6.2/icon.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added website/src/assets/0.6.2/live-streaming.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added website/src/assets/0.6.2/pr-approval.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
296 changes: 296 additions & 0 deletions website/src/content/docs/0.6.2/architecture.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,296 @@
---
slug: 0.6.2/architecture
title: Architecture
description: One-page overview of how the bot is structured and how a review flows through it.
---




One-page overview of how the bot is structured and how a review flows through it.

ThrillhouseBot is a Quarkus application that runs as a GitHub App. A webhook
arrives when a pull request changes, the bot builds a review with an
OpenAI-compatible model, and it posts the result back as a PR review plus a
check run. A dashboard streams what is happening live.

## Components

```mermaid
flowchart TB
subgraph GH[GitHub.com]
IN[PR push, /review, @mention]
OUT[PR reviews Β· check runs Β· comments]
end

subgraph BOT[ThrillhouseBot Β· Quarkus]
WH[webhook/<br/>WebhookController]
RO[review/<br/>ReviewOrchestrator]
AI[review/ai/<br/>AiReviewService]
GHC[github/<br/>REST clients]
DB[dashboard/ + frontend/]

WH --> RO --> AI
RO --> GHC
AI -.->|live tokens / review.batch| DB
end

IN -->|POST /api/webhook<br/>HMAC-verified| WH
GHC --> OUT
```

The `github/` clients wrap the GitHub REST surface the bot uses: installation
tokens, pull diffs and prior reviews, check runs, PR reviews with inline
comments, issue comments, and the instructions-file fallback chain
(`.github/thrillhousebot.md`, `.github/copilot-instructions.md`, `CLAUDE.md`,
`AGENTS.md`, `AGENT.md`).

## Request flow

```mermaid
flowchart TD
GH[GitHub: PR opened / synced / comment] -->|webhook| WH[webhook/]
WH -->|verify HMAC, filters, rate limit, πŸ‘€ ack| RO[review/ ReviewOrchestrator]
RO -->|fetch diff, instructions, prior findings| GHC[github/ API clients]
RO -->|budget plan Β· stream or batch| AI[review/ai/ LangChain4j]
AI -->|parse findings| RO
RO -->|verify findings β€” 2nd AI call per batch, on by default| AI
RO -->|post review + check run| GHC --> GH
AI -.->|live tokens or review.batch| DB[dashboard/ broadcaster]
DB -->|WebSocket| FE[frontend/ Next.js UI]
RO -->|persist session, cost, tokens| PG[(H2 / PostgreSQL)]
AI -->|traces, token & cost metrics| OT[(OpenTelemetry)]
```

Automatic triggers (`pull_request` opened / synchronize, and similar) are subject
to `AUTO_REVIEW_MIN_INTERVAL`: if the same PR was auto-reviewed too recently,
the webhook path skips the review silently. Manual `/review` always bypasses that
window. Slash and mention **commands** get a best-effort πŸ‘€ reaction before
pause/authorization; conversational `@thrillhousebot` mentions (no command word)
are answered without a reaction.

## Review lifecycle

### First review (PR opened)

```mermaid
sequenceDiagram
actor Dev
participant GH as GitHub
participant TB as ThrillhouseBot
participant AI as AI Provider

Dev->>GH: git push (PR opened)
GH->>TB: POST /api/webhook (pull_request: opened)

Note over TB: Verify HMAC β†’ JWT β†’ install token
Note over TB: Auto-review rate limit (skip if within AUTO_REVIEW_MIN_INTERVAL)

TB->>GH: POST check-runs β†’ status: queued
TB->>GH: PATCH check-run β†’ status: in_progress

par Fetch context
TB->>GH: GET /pulls/{pr}/files (diff)
TB->>GH: GET /compare/{base}...{head} (regression context)
end

TB->>GH: GET /pulls/{pr}/reviews (check if already reviewed this SHA)
GH-->>TB: no prior reviews β†’ first run

Note over TB: DiffBudgetPlanner β€” single-call or token-budgeted batches

alt Diff fits one call
TB->>AI: POST chat (diff + base comparison + review prompt) β€” live tokens to dashboard
AI-->>TB: findings + risk levels + suggestions
opt Findings found and REVIEW_VERIFIER_ENABLED (default)
TB->>AI: POST chat (re-check each finding against the diff)
AI-->>TB: confirmed / downgraded / dropped findings
end
else Large diff β€” multi-call
loop Each batch in parallel (up to REVIEW_MAX_AI_CALLS βˆ’ 1)
TB-->>TB: review.batch progress (no per-token stream)
TB->>AI: POST chat (batch diff)
AI-->>TB: batch findings
opt Findings and verifier on
TB->>AI: POST chat (verify batch findings)
end
end
TB->>AI: POST chat (summary rollup of aggregated findings)
AI-->>TB: PR-level summary + previous-findings status
Note over TB: Any files that still won't fit are disclosed by name
end

alt AI fails
TB->>GH: PATCH check-run β†’ conclusion: failure
TB->>GH: POST comment: retry hint (no internal details)
else AI succeeds + issues found
TB->>GH: POST PR review (REQUEST_CHANGES or COMMENT) with inline suggestions
TB->>GH: PATCH check-run β†’ conclusion: failure (critical/high) or neutral
TB->>GH: POST comment: PR summary (risk table + key findings)
else AI succeeds + zero issues
TB->>GH: POST PR review β†’ APPROVE (no body)
TB->>GH: PATCH check-run β†’ conclusion: success
TB->>GH: POST comment: PR summary (celebration inside)
end
```

### Follow-up review (new push)

```mermaid
sequenceDiagram
actor Dev
participant GH as GitHub
participant TB as ThrillhouseBot
participant AI as AI Provider

Dev->>GH: git push (PR synchronize)
GH->>TB: POST /api/webhook (pull_request: synchronize)

Note over TB: Verify β†’ auth β†’ rate limit β†’ create check run (in_progress)

par Fetch context
TB->>GH: GET /pulls/{pr}/files (diff)
TB->>GH: GET /compare/{base}...{head}
and Fetch prior review
TB->>GH: GET /pulls/{pr}/reviews (find ThrillhouseBot's last review)
GH-->>TB: previous findings + thread status
end

Note over TB: Prompt includes diff + prior findings + "check if each was addressed"
Note over TB: Same single-call or map-reduce path as first review

TB->>AI: POST chat (one or more review calls Β± verifier Β± summary)
AI-->>TB: resolved / unresolved / new findings

alt AI fails
Note over TB: Same sanitized error path as first review
else AI succeeds
TB->>GH: POST PR review (suggestions for unresolved + new issues)
TB->>GH: PATCH check-run β†’ conclusion based on risk
Note over TB: No summary comment on follow-up (only on first run)
end
```

### Manual trigger (`/review` or `@Thrillhousebot review`)

```mermaid
sequenceDiagram
actor Dev
participant GH as GitHub
participant TB as ThrillhouseBot

Dev->>GH: Comments "/review"
GH->>TB: POST /api/webhook (issue_comment: created)

Note over TB: Verify β†’ parse trigger β†’ πŸ‘€ ack (bounded wait) β†’ auth
Note over TB: Manual /review bypasses AUTO_REVIEW_MIN_INTERVAL
Note over TB: Fetch diff, compare, and prior reviews
Note over TB: Full re-review even if this SHA was already reviewed
```

### Conversational reply (`@thrillhousebot` mention)

```mermaid
sequenceDiagram
actor Dev
participant GH as GitHub
participant TB as ThrillhouseBot

Dev->>GH: Mentions @thrillhousebot (in a PR thread or finding reply)
GH->>TB: POST /api/webhook (pull_request_review_comment or issue_comment: created)

Note over TB: Bot-loop guard β†’ mention detected (no command) β†’ ACK 200
Note over TB: No πŸ‘€ reaction β€” conversational mentions are answered, not reacted to
Note over TB: Async on review executor: authorize (write access)
Note over TB: Build threaded prompt (finding + diff hunk + thread)
TB->>GH: POST reply in the review thread (or a PR comment)
```

## Packages

| Package | Responsibility | Notable classes |
|---|---|---|
| `webhook/` | Receives GitHub events, verifies the HMAC signature, decides whether an event triggers a review (trigger filters, per-PR pause state, auto-review rate limit), acks slash/mention commands with πŸ‘€, runs the comment commands (`/help`, `/summary`, `/describe`, `/changelog`, `/add-docs`, `/improve`, `/generate-tests`, `/resolve`, `/pause`, `/resume`), and schedules finding-feedback capture on review-thread replies | `WebhookController`, `WebhookVerifier`, `TriggerDetector`, `ReviewTriggerFilter`, `AckReactionService`, `CommentCommandService`, `PrPauseService` |
| `review/` | Orchestrates a review: plans the token budget and the per-review spend ceiling, calls the AI layer (single-call or map-reduce), maps findings to a risk level and review state, re-checks a maintainer's decline against the reviewed code, writes the summary comment, optionally labels the PR, answers maintainer replies/mentions in PR threads, and persists maintainer finding feedback (πŸ‘/πŸ‘Ž / reply heuristics) for a future learnings pipeline | `ReviewOrchestrator`, `ReviewDispatcher`, `DiffBudgetPlanner`, `FindingPipeline`, `AutoReviewRateLimiter`, `ReviewDiffFormatter`, `FollowUpAnalyzer`, `FindingFeedbackCaptureService`, `FindingFeedbackService`, `PrSummaryGenerator`, `PrLabeler`, `MaintainerReplyService`, `MaintainerReplyDispatcher`, `PrImprovementService`, `PatchCoverage`, `ConfigKeyContextResolver`, `RebuttalContradiction`, `SummarySurfaceDeduplicator`, `VerdictBuilder` |
| `review/ai/` | The LangChain4j layer: streams or batches model responses, parses findings, runs a second pass to verify them, applies generation/reasoning customizers, and writes conversational replies | `PrReviewer`, `AiReviewService`, `ChatModelCustomizers`, `FindingVerifier`, `FindingVerificationService`, `ReviewResponseParser`, `ReplyAssistant`, `TruncatedResponseSalvager`, `FindingVerifierPrompts` |
| `github/` | Talks to the GitHub REST and GraphQL APIs: app auth, pull requests, reviews, check runs, comments, labels, reactions (create + list), and reading the repo instructions file | `GitHubAuthClient`, `GitHubReviewClient`, `GitHubCheckRunClient`, `GitHubLabelClient`, `GitHubReactionClient`, `InstructionsResolver`, `GitHubWriteRetry` |
| `dashboard/` | The live UI backend: OAuth login (in-memory sessions), WebSocket broadcaster (`review.stream` / `review.batch`), review session persistence, and finding-feedback aggregates | `AuthResource`, `DashboardSessionStore`, `SessionEventBroadcaster`, `ReviewSessionRepository`, `DashboardResource` |
| `config/` | Wiring: the outbound HTTP client, the review thread pool, typed config, active-model settings (caps, generation params), fail-fast startup validation, and the shared bot-identity used to recognize the bot's own activity | `HttpClientProducer`, `ReviewExecutorProducer`, `ThrillhouseConfig`, `ActiveModelSettings`, `StartupConfigValidator`, `BotIdentity` |
| `frontend/` | The Next.js dashboard, built to a static export and served by Quarkus | β€” |

## Notes

PR reviews carry inline comments and suggestions; check runs carry pass/fail
status for branch protection (no inline annotations on the check run itself).

**AI call budget** β€” a review that reports findings makes **two** model calls
by default: the review call plus a skeptical verification pass
(`FindingVerifier`) that re-sends the diff and each candidate finding, dropping
or downgrading what it can't confirm. It fails open β€” a verifier error keeps
the original findings, so a broken verifier can never block a review. Under
token-aware budgeting on large PRs this becomes N batch review calls + N
per-batch verification calls + one summary call. `REVIEW_VERIFIER_ENABLED=false`
skips only the AI pass (a deterministic hedging-language guard still runs) and
trades cost for more false positives. Expect two model spans per flagged
single-call review (or N+N+1 under budgeting) in the traces and in the
dashboard's session totals. Multi-call reviews do not stream tokens to the
dashboard; they emit `review.batch` progress events instead. Batches run
concurrently on virtual threads; a failed batch is retried once after the
parallel pass completes.

**Cost ceiling** β€” `REVIEW_MAX_TOKENS_PER_REVIEW` bounds the tokens one review may
spend across every call it makes, counting retries, the verifier and the summary.
Once reached, remaining batches are disclosed as not reviewed by name and the
summary degrades to counts rather than making further calls. `0`, the default,
leaves it unbounded. The summary, the verifier and maintainer replies run on a
separate `concise` model binding with its own response cap
(`REVIEW_CONCISE_MAX_OUTPUT_TOKENS`) and its own reasoning effort, so they never
share a cap sized for batch review output.

**Coverage honesty** β€” a file the review never read does not pass silently. A
file GitHub reported with changes but no patch text, a file that did not fit any
batch, and a file whose batch call failed are each disclosed by name and withhold
APPROVE. A response the model cut at its length cap keeps the findings that
completed before the cut rather than being discarded whole.

**Patch coverage as review context** β€” when a PR's CI publishes a coverage
report, `PatchCoverage` reads the changed lines it does not cover and gives them
to the review, so new code with no test behind it can be named as such. Off
unless `REVIEW_PATCH_COVERAGE_ENABLED` is set.

**Repository-supplied configuration** β€” `.github/thrillhousebot.yml` carries a
repository's own ignore globs and path-scoped review instructions, read from the
default branch and cached for five minutes. Ignore globs are additive to the
deployment list; a repository can narrow its own review scope but cannot restore
a file the deployment excludes. Every failure mode (missing file, invalid YAML,
unexpected shape, uncompilable glob) is logged and skipped, leaving the
deployment configuration in force.

**Write pacing** β€” content-creating GitHub calls are spaced process-wide by
`GITHUB_WRITE_MIN_INTERVAL` so a burst of comments never reaches the secondary
rate limit in the first place, with `GITHUB_WRITE_MAX_WAIT` capping how long any
one caller waits.

Each AI call is bounded by `AI_TIMEOUT` (LangChain4j) and
`thrillhousebot.review.ai-timeout-seconds`. Cost and token metrics come from
OpenTelemetry. OAuth login sessions are opaque IDs in cookies with tokens kept
server-side; review history persists in the database. Maintainer finding
feedback (reactions and reply heuristics) is documented in
[Finding feedback](https://devops-thiago.github.io/ThrillhouseBot/feedback/)
(source: [docs/FEEDBACK.md](https://github.com/devops-thiago/ThrillhouseBot/blob/main/docs/FEEDBACK.md)). See
[SECURITY.md](https://github.com/devops-thiago/ThrillhouseBot/blob/main/SECURITY.md)
for the reporting process.

## Adding an AI provider

There is no provider-specific code. The model is reached through LangChain4j's
OpenAI-compatible client, so a new provider is configuration: point `AI_BASE_URL`
and `AI_MODEL` at it. Add a `thrillhousebot.ai.pricing.<model>.*` pair for cost
tracking (without it the bot warns once and flags sessions as "no pricing"
instead of `$0`). Optionally set `thrillhousebot.ai.models.<model>.*` for the
model's input cap and generation parameters, and `AI_REASONING_ENABLED` /
`AI_REASONING_EFFORT` when the model supports reasoning. See the
[provider table](https://devops-thiago.github.io/ThrillhouseBot/providers/) and
the [configuration reference](https://devops-thiago.github.io/ThrillhouseBot/configuration/).


Loading
Loading