diff --git a/CHANGELOG.md b/CHANGELOG.md index 17c7120b..3aa0d3ad 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,24 @@ All notable changes to ThrillhouseBot. ## [Unreleased] +## [0.6.3] — 2026-08-15 + +Follow-ups from the round-7 dogfood corpus, on two surfaces: the GitHub write +path, and what a review says about its own work. A comment GitHub throttles is +now outlasted and its refusal explained, a finding whose inline comment cannot +land keeps a working review thread, and a review can no longer claim coverage +or closures its audit never made. The docs site also deploys itself on +release. No configuration changes; upgrading is a redeploy. + +### Fixed + +- **A comment GitHub refuses is outlasted, and the refusal reason reaches the log** (#722): the content-creation secondary limit was measured blocking for 72 seconds while the write retry's budget spanned 60, and both ways of deriving a delay without a `Retry-After` undershot it — the linear fallback summed to 30 seconds, and a stale `x-ratelimit-reset` yielded zero, spending every attempt while GitHub was still refusing. The retry now makes four attempts and floors a derived delay at 30 seconds during a content-creation block; an explicit `Retry-After` still wins outright, and a genuinely exhausted primary window keeps its own reset instant. Every rejection now logs GitHub's status and message at warning level, so a throttle is distinguishable from a rejected position — the round that motivated this left 29 rejections undiagnosable and blamed line anchoring for them. When the with-suggestion and without-suggestion attempts fail differently, both reasons are kept +- **A finding whose inline comment cannot land posts a file-level thread, not a bare bullet** (#712): the bullet carried no code context, no suggestion block and no review thread, and everything thread-dependent died with it — the finding could not be declined, a status note had nowhere to land, and clearing it acknowledged a count rather than a name. The suspected anchoring defect was ruled out: the same file and line posted successfully at the same commit, which is what pointed to the throttle above +- **An overturned decline is disclosed on every review body, and a round names the findings it closed** (#713, #714): the note explaining that a decline was overturned on diff evidence rode only the no-new-findings body, so a round that also raised a finding dropped it, and a maintainer who wrote a considered decline could not tell whether it was read, rebutted, or missed. The note now rides the with-findings bodies too, above the partial-coverage banner. The resolved tally likewise names each finding it closed instead of reporting a bare count +- **Verification coverage counts only the verdicts the audit acted on** (#710): the coverage record asked whether some verdict carried a finding's id, but `apply()` acts only on `confirmed`, `downgraded` and `rejected` — a blank or unrecognized verdict falls open and the finding posts unscreened. Such findings were counted as screened anyway, so a review could claim full verification over a set it never ruled on. One decision normalizer now feeds both the switch and the count, and an unrecognized verdict counts as unverified, which errs toward an over-cautious clause rather than silence +- **The verifier sees the PR description it judges description-gap findings against** (#711, first part): a finding weighing the author's stated intent against the code had half its claim in material the verifier never received, and the verifier's own prompt tells it to reject a claim whose material is missing — so survival came down to whether the model noticed the absence. Five planted description-versus-code mismatches split three kept, two rejected, on identical grounds. The verifier now receives the PR title and description under the same untrusted-data fencing the reviewer uses +- **The docs site deploys itself on release** (#717): the `release: published` trigger in `docs.yml` has fired zero times in the repository's history, because the release is created with the workflow's own `GITHUB_TOKEN` and GitHub starts no workflows for events that token raises — every site deploy through v0.6.2 was manual. `release.yml` now dispatches the docs build against the release tag once the release exists, gated on `update_latest` so a patch cut on an older line does not republish the site; the job fails loudly when the dispatch is refused + ## [0.6.2] — 2026-08-14 Follow-ups to the review threads on 0.6.1, plus the first piece of the release diff --git a/pom.xml b/pom.xml index ffb20671..3f61bb9b 100644 --- a/pom.xml +++ b/pom.xml @@ -8,7 +8,7 @@ dev.thiagogonzaga.thrillhousebot thrillhousebot - 0.6.3-SNAPSHOT + 0.6.3 3.15.0 diff --git a/website/src/assets/0.6.2/icon.png b/website/src/assets/0.6.2/icon.png new file mode 100644 index 00000000..e523b074 Binary files /dev/null and b/website/src/assets/0.6.2/icon.png differ diff --git a/website/src/assets/0.6.2/live-streaming.png b/website/src/assets/0.6.2/live-streaming.png new file mode 100644 index 00000000..d6728e7d Binary files /dev/null and b/website/src/assets/0.6.2/live-streaming.png differ diff --git a/website/src/assets/0.6.2/pr-approval.png b/website/src/assets/0.6.2/pr-approval.png new file mode 100644 index 00000000..1767740c Binary files /dev/null and b/website/src/assets/0.6.2/pr-approval.png differ diff --git a/website/src/content/docs/0.6.2/architecture.md b/website/src/content/docs/0.6.2/architecture.md new file mode 100644 index 00000000..f7f4144c --- /dev/null +++ b/website/src/content/docs/0.6.2/architecture.md @@ -0,0 +1,296 @@ +--- +slug: 0.6.2/architecture +title: Architecture +description: One-page overview of how the bot is structured and how a review flows through it. +--- + + + + +One-page overview of how the bot is structured and how a review flows through it. + +ThrillhouseBot is a Quarkus application that runs as a GitHub App. A webhook +arrives when a pull request changes, the bot builds a review with an +OpenAI-compatible model, and it posts the result back as a PR review plus a +check run. A dashboard streams what is happening live. + +## Components + +```mermaid +flowchart TB + subgraph GH[GitHub.com] + IN[PR push, /review, @mention] + OUT[PR reviews · check runs · comments] + end + + subgraph BOT[ThrillhouseBot · Quarkus] + WH[webhook/
WebhookController] + RO[review/
ReviewOrchestrator] + AI[review/ai/
AiReviewService] + GHC[github/
REST clients] + DB[dashboard/ + frontend/] + + WH --> RO --> AI + RO --> GHC + AI -.->|live tokens / review.batch| DB + end + + IN -->|POST /api/webhook
HMAC-verified| WH + GHC --> OUT +``` + +The `github/` clients wrap the GitHub REST surface the bot uses: installation +tokens, pull diffs and prior reviews, check runs, PR reviews with inline +comments, issue comments, and the instructions-file fallback chain +(`.github/thrillhousebot.md`, `.github/copilot-instructions.md`, `CLAUDE.md`, +`AGENTS.md`, `AGENT.md`). + +## Request flow + +```mermaid +flowchart TD + GH[GitHub: PR opened / synced / comment] -->|webhook| WH[webhook/] + WH -->|verify HMAC, filters, rate limit, 👀 ack| RO[review/ ReviewOrchestrator] + RO -->|fetch diff, instructions, prior findings| GHC[github/ API clients] + RO -->|budget plan · stream or batch| AI[review/ai/ LangChain4j] + AI -->|parse findings| RO + RO -->|verify findings — 2nd AI call per batch, on by default| AI + RO -->|post review + check run| GHC --> GH + AI -.->|live tokens or review.batch| DB[dashboard/ broadcaster] + DB -->|WebSocket| FE[frontend/ Next.js UI] + RO -->|persist session, cost, tokens| PG[(H2 / PostgreSQL)] + AI -->|traces, token & cost metrics| OT[(OpenTelemetry)] +``` + +Automatic triggers (`pull_request` opened / synchronize, and similar) are subject +to `AUTO_REVIEW_MIN_INTERVAL`: if the same PR was auto-reviewed too recently, +the webhook path skips the review silently. Manual `/review` always bypasses that +window. Slash and mention **commands** get a best-effort 👀 reaction before +pause/authorization; conversational `@thrillhousebot` mentions (no command word) +are answered without a reaction. + +## Review lifecycle + +### First review (PR opened) + +```mermaid +sequenceDiagram + actor Dev + participant GH as GitHub + participant TB as ThrillhouseBot + participant AI as AI Provider + + Dev->>GH: git push (PR opened) + GH->>TB: POST /api/webhook (pull_request: opened) + + Note over TB: Verify HMAC → JWT → install token + Note over TB: Auto-review rate limit (skip if within AUTO_REVIEW_MIN_INTERVAL) + + TB->>GH: POST check-runs → status: queued + TB->>GH: PATCH check-run → status: in_progress + + par Fetch context + TB->>GH: GET /pulls/{pr}/files (diff) + TB->>GH: GET /compare/{base}...{head} (regression context) + end + + TB->>GH: GET /pulls/{pr}/reviews (check if already reviewed this SHA) + GH-->>TB: no prior reviews → first run + + Note over TB: DiffBudgetPlanner — single-call or token-budgeted batches + + alt Diff fits one call + TB->>AI: POST chat (diff + base comparison + review prompt) — live tokens to dashboard + AI-->>TB: findings + risk levels + suggestions + opt Findings found and REVIEW_VERIFIER_ENABLED (default) + TB->>AI: POST chat (re-check each finding against the diff) + AI-->>TB: confirmed / downgraded / dropped findings + end + else Large diff — multi-call + loop Each batch in parallel (up to REVIEW_MAX_AI_CALLS − 1) + TB-->>TB: review.batch progress (no per-token stream) + TB->>AI: POST chat (batch diff) + AI-->>TB: batch findings + opt Findings and verifier on + TB->>AI: POST chat (verify batch findings) + end + end + TB->>AI: POST chat (summary rollup of aggregated findings) + AI-->>TB: PR-level summary + previous-findings status + Note over TB: Any files that still won't fit are disclosed by name + end + + alt AI fails + TB->>GH: PATCH check-run → conclusion: failure + TB->>GH: POST comment: retry hint (no internal details) + else AI succeeds + issues found + TB->>GH: POST PR review (REQUEST_CHANGES or COMMENT) with inline suggestions + TB->>GH: PATCH check-run → conclusion: failure (critical/high) or neutral + TB->>GH: POST comment: PR summary (risk table + key findings) + else AI succeeds + zero issues + TB->>GH: POST PR review → APPROVE (no body) + TB->>GH: PATCH check-run → conclusion: success + TB->>GH: POST comment: PR summary (celebration inside) + end +``` + +### Follow-up review (new push) + +```mermaid +sequenceDiagram + actor Dev + participant GH as GitHub + participant TB as ThrillhouseBot + participant AI as AI Provider + + Dev->>GH: git push (PR synchronize) + GH->>TB: POST /api/webhook (pull_request: synchronize) + + Note over TB: Verify → auth → rate limit → create check run (in_progress) + + par Fetch context + TB->>GH: GET /pulls/{pr}/files (diff) + TB->>GH: GET /compare/{base}...{head} + and Fetch prior review + TB->>GH: GET /pulls/{pr}/reviews (find ThrillhouseBot's last review) + GH-->>TB: previous findings + thread status + end + + Note over TB: Prompt includes diff + prior findings + "check if each was addressed" + Note over TB: Same single-call or map-reduce path as first review + + TB->>AI: POST chat (one or more review calls ± verifier ± summary) + AI-->>TB: resolved / unresolved / new findings + + alt AI fails + Note over TB: Same sanitized error path as first review + else AI succeeds + TB->>GH: POST PR review (suggestions for unresolved + new issues) + TB->>GH: PATCH check-run → conclusion based on risk + Note over TB: No summary comment on follow-up (only on first run) + end +``` + +### Manual trigger (`/review` or `@Thrillhousebot review`) + +```mermaid +sequenceDiagram + actor Dev + participant GH as GitHub + participant TB as ThrillhouseBot + + Dev->>GH: Comments "/review" + GH->>TB: POST /api/webhook (issue_comment: created) + + Note over TB: Verify → parse trigger → 👀 ack (bounded wait) → auth + Note over TB: Manual /review bypasses AUTO_REVIEW_MIN_INTERVAL + Note over TB: Fetch diff, compare, and prior reviews + Note over TB: Full re-review even if this SHA was already reviewed +``` + +### Conversational reply (`@thrillhousebot` mention) + +```mermaid +sequenceDiagram + actor Dev + participant GH as GitHub + participant TB as ThrillhouseBot + + Dev->>GH: Mentions @thrillhousebot (in a PR thread or finding reply) + GH->>TB: POST /api/webhook (pull_request_review_comment or issue_comment: created) + + Note over TB: Bot-loop guard → mention detected (no command) → ACK 200 + Note over TB: No 👀 reaction — conversational mentions are answered, not reacted to + Note over TB: Async on review executor: authorize (write access) + Note over TB: Build threaded prompt (finding + diff hunk + thread) + TB->>GH: POST reply in the review thread (or a PR comment) +``` + +## Packages + +| Package | Responsibility | Notable classes | +|---|---|---| +| `webhook/` | Receives GitHub events, verifies the HMAC signature, decides whether an event triggers a review (trigger filters, per-PR pause state, auto-review rate limit), acks slash/mention commands with 👀, runs the comment commands (`/help`, `/summary`, `/describe`, `/changelog`, `/add-docs`, `/improve`, `/generate-tests`, `/resolve`, `/pause`, `/resume`), and schedules finding-feedback capture on review-thread replies | `WebhookController`, `WebhookVerifier`, `TriggerDetector`, `ReviewTriggerFilter`, `AckReactionService`, `CommentCommandService`, `PrPauseService` | +| `review/` | Orchestrates a review: plans the token budget and the per-review spend ceiling, calls the AI layer (single-call or map-reduce), maps findings to a risk level and review state, re-checks a maintainer's decline against the reviewed code, writes the summary comment, optionally labels the PR, answers maintainer replies/mentions in PR threads, and persists maintainer finding feedback (👍/👎 / reply heuristics) for a future learnings pipeline | `ReviewOrchestrator`, `ReviewDispatcher`, `DiffBudgetPlanner`, `FindingPipeline`, `AutoReviewRateLimiter`, `ReviewDiffFormatter`, `FollowUpAnalyzer`, `FindingFeedbackCaptureService`, `FindingFeedbackService`, `PrSummaryGenerator`, `PrLabeler`, `MaintainerReplyService`, `MaintainerReplyDispatcher`, `PrImprovementService`, `PatchCoverage`, `ConfigKeyContextResolver`, `RebuttalContradiction`, `SummarySurfaceDeduplicator`, `VerdictBuilder` | +| `review/ai/` | The LangChain4j layer: streams or batches model responses, parses findings, runs a second pass to verify them, applies generation/reasoning customizers, and writes conversational replies | `PrReviewer`, `AiReviewService`, `ChatModelCustomizers`, `FindingVerifier`, `FindingVerificationService`, `ReviewResponseParser`, `ReplyAssistant`, `TruncatedResponseSalvager`, `FindingVerifierPrompts` | +| `github/` | Talks to the GitHub REST and GraphQL APIs: app auth, pull requests, reviews, check runs, comments, labels, reactions (create + list), and reading the repo instructions file | `GitHubAuthClient`, `GitHubReviewClient`, `GitHubCheckRunClient`, `GitHubLabelClient`, `GitHubReactionClient`, `InstructionsResolver`, `GitHubWriteRetry` | +| `dashboard/` | The live UI backend: OAuth login (in-memory sessions), WebSocket broadcaster (`review.stream` / `review.batch`), review session persistence, and finding-feedback aggregates | `AuthResource`, `DashboardSessionStore`, `SessionEventBroadcaster`, `ReviewSessionRepository`, `DashboardResource` | +| `config/` | Wiring: the outbound HTTP client, the review thread pool, typed config, active-model settings (caps, generation params), fail-fast startup validation, and the shared bot-identity used to recognize the bot's own activity | `HttpClientProducer`, `ReviewExecutorProducer`, `ThrillhouseConfig`, `ActiveModelSettings`, `StartupConfigValidator`, `BotIdentity` | +| `frontend/` | The Next.js dashboard, built to a static export and served by Quarkus | — | + +## Notes + +PR reviews carry inline comments and suggestions; check runs carry pass/fail +status for branch protection (no inline annotations on the check run itself). + +**AI call budget** — a review that reports findings makes **two** model calls +by default: the review call plus a skeptical verification pass +(`FindingVerifier`) that re-sends the diff and each candidate finding, dropping +or downgrading what it can't confirm. It fails open — a verifier error keeps +the original findings, so a broken verifier can never block a review. Under +token-aware budgeting on large PRs this becomes N batch review calls + N +per-batch verification calls + one summary call. `REVIEW_VERIFIER_ENABLED=false` +skips only the AI pass (a deterministic hedging-language guard still runs) and +trades cost for more false positives. Expect two model spans per flagged +single-call review (or N+N+1 under budgeting) in the traces and in the +dashboard's session totals. Multi-call reviews do not stream tokens to the +dashboard; they emit `review.batch` progress events instead. Batches run +concurrently on virtual threads; a failed batch is retried once after the +parallel pass completes. + +**Cost ceiling** — `REVIEW_MAX_TOKENS_PER_REVIEW` bounds the tokens one review may +spend across every call it makes, counting retries, the verifier and the summary. +Once reached, remaining batches are disclosed as not reviewed by name and the +summary degrades to counts rather than making further calls. `0`, the default, +leaves it unbounded. The summary, the verifier and maintainer replies run on a +separate `concise` model binding with its own response cap +(`REVIEW_CONCISE_MAX_OUTPUT_TOKENS`) and its own reasoning effort, so they never +share a cap sized for batch review output. + +**Coverage honesty** — a file the review never read does not pass silently. A +file GitHub reported with changes but no patch text, a file that did not fit any +batch, and a file whose batch call failed are each disclosed by name and withhold +APPROVE. A response the model cut at its length cap keeps the findings that +completed before the cut rather than being discarded whole. + +**Patch coverage as review context** — when a PR's CI publishes a coverage +report, `PatchCoverage` reads the changed lines it does not cover and gives them +to the review, so new code with no test behind it can be named as such. Off +unless `REVIEW_PATCH_COVERAGE_ENABLED` is set. + +**Repository-supplied configuration** — `.github/thrillhousebot.yml` carries a +repository's own ignore globs and path-scoped review instructions, read from the +default branch and cached for five minutes. Ignore globs are additive to the +deployment list; a repository can narrow its own review scope but cannot restore +a file the deployment excludes. Every failure mode (missing file, invalid YAML, +unexpected shape, uncompilable glob) is logged and skipped, leaving the +deployment configuration in force. + +**Write pacing** — content-creating GitHub calls are spaced process-wide by +`GITHUB_WRITE_MIN_INTERVAL` so a burst of comments never reaches the secondary +rate limit in the first place, with `GITHUB_WRITE_MAX_WAIT` capping how long any +one caller waits. + +Each AI call is bounded by `AI_TIMEOUT` (LangChain4j) and +`thrillhousebot.review.ai-timeout-seconds`. Cost and token metrics come from +OpenTelemetry. OAuth login sessions are opaque IDs in cookies with tokens kept +server-side; review history persists in the database. Maintainer finding +feedback (reactions and reply heuristics) is documented in +[Finding feedback](https://devops-thiago.github.io/ThrillhouseBot/feedback/) +(source: [docs/FEEDBACK.md](https://github.com/devops-thiago/ThrillhouseBot/blob/main/docs/FEEDBACK.md)). See +[SECURITY.md](https://github.com/devops-thiago/ThrillhouseBot/blob/main/SECURITY.md) +for the reporting process. + +## Adding an AI provider + +There is no provider-specific code. The model is reached through LangChain4j's +OpenAI-compatible client, so a new provider is configuration: point `AI_BASE_URL` +and `AI_MODEL` at it. Add a `thrillhousebot.ai.pricing..*` pair for cost +tracking (without it the bot warns once and flags sessions as "no pricing" +instead of `$0`). Optionally set `thrillhousebot.ai.models..*` for the +model's input cap and generation parameters, and `AI_REASONING_ENABLED` / +`AI_REASONING_EFFORT` when the model supports reasoning. See the +[provider table](https://devops-thiago.github.io/ThrillhouseBot/providers/) and +the [configuration reference](https://devops-thiago.github.io/ThrillhouseBot/configuration/). + + diff --git a/website/src/content/docs/0.6.2/commands.md b/website/src/content/docs/0.6.2/commands.md new file mode 100644 index 00000000..46cfcb42 --- /dev/null +++ b/website/src/content/docs/0.6.2/commands.md @@ -0,0 +1,96 @@ +--- +slug: 0.6.2/commands +title: Commands +description: Drive the bot directly from a PR with comment commands. +--- + + + +Drive the bot directly from a PR by commenting one of these. Each also accepts the +mention form, e.g. `@Thrillhousebot review`. The bot acknowledges every command +instantly with a 👀 reaction on your comment while the work runs in the background; +a conversational `@thrillhousebot` mention (no command word) gets an answer instead, +not a reaction. + +| Command | What it does | Access | +|---|---|---| +| `/help` | List the available commands | anyone | +| `/review` | Run (or re-run) a full review of the PR | write | +| `/summary` | Post the PR summary if it isn't already on the PR — regenerates it if the comment was deleted, otherwise no-op | write | +| `/describe` | Suggest an improved PR title and description generated from the diff, as a comment to copy in (never overwrites the PR) | write | +| `/changelog` | Draft a CHANGELOG entry for the PR from the diff (Added/Changed/Fixed/Security…), as a comment to copy into `CHANGELOG.md` (never commits) | write | +| `/add-docs` | Generate docstrings/inline docs for the symbols changed in the PR, posted as committable suggestions (or a note with the drafted docs when a multi-line declaration can't be pinned to a single diff hunk) | write | +| `/improve` | Run a whole-PR improvement pass over the diff and post the improvements as committable suggestions (with copy-paste blocks for the ones that can't be pinned to the diff) | write | +| `/generate-tests` | Propose unit tests for the code the PR changed, as a comment with one ready-to-paste code block per test file (never commits) | write | +| `/resolve` | Resolve ThrillhouseBot's outstanding finding threads on the PR | write | +| `/pause` | Silence the bot on the PR | write | +| `/resume` | Re-enable the bot on a paused PR | write | +| `@thrillhousebot resolved :` | Close a previous finding that has no review thread to reply on, so it stops holding approval (see **Clearing a finding with no thread** under Configuration) | write | + +**Access** — every slash command except `/help` requires the commenter to hold write access +to the repository, or to be named in `THRILLHOUSEBOT_REVIEW_MANUAL_TRIGGER_ALLOWED_LOGINS`, +since reviews spend the operator's AI budget. The allowlist covers the slash commands only: +the `@thrillhousebot resolved` directive always requires write access, as described below. + +**`@thrillhousebot resolved`** — a directive, not a slash command: it has no `/resolved` +form, and it is read by the *next* review rather than acted on immediately. The bot replies +straight away to say what that review will evaluate — and, when the comment names no +`path:line` at all, to say plainly that nothing will be cleared, so a mistyped locator shows +up immediately instead of as a review that changes nothing. It is a statement, never a +question: `@thrillhousebot resolved?` is asking, not deciding, and clears nothing. Do not +confuse it with `/resolve`, which resolves GitHub review threads and does nothing to a +finding that never opened one. Its access check is the commenter's GitHub +`author_association` on that comment — `OWNER`, `MEMBER` or `COLLABORATOR` — and +`THRILLHOUSEBOT_REVIEW_MANUAL_TRIGGER_ALLOWED_LOGINS` does *not* extend it. + +**Pause** — while a PR is paused, ThrillhouseBot skips automatic reviews on new commits, +ignores `/review`, `/summary`, `/describe`, `/changelog`, `/add-docs`, `/improve` and +`/generate-tests`, and does not answer `@thrillhousebot` mentions (it replies once to say it is +paused). `/resume` lifts the pause. +`/help` and `/resolve` keep working while paused. + +**`/describe` and `/changelog`** — both read the whole change set rather than the first +`REVIEW_MAX_DIFF_LINES` of it. The changed files are packed into batches that each fit +`REVIEW_MAX_INPUT_TOKENS`, one model call per batch, and the per-batch results are then reduced +to a single answer: `/describe` composes the partial descriptions into one coherent title and +description, `/changelog` merges the candidate entries into one entry. That reduce step costs one +extra model call, reserved out of `REVIEW_MAX_AI_CALLS` and spent only when the PR actually needed +more than one batch, so a run never exceeds the same ceiling as one review. Any file the budget +could not cover is named in a partial-coverage note under the suggestion. + +**`/add-docs`** — on demand, the bot reads the diff and proposes documentation comments for +the public symbols changed in the PR, honoring the repository instructions and each file's +language. Each suggestion is a committable `suggestion` block placed on the symbol's +declaration (spanning the whole signature when it wraps), so it only inserts docs without +rewriting code. When a multi-line declaration can't be pinned to a single diff hunk, the bot +posts a note with the drafted docs to add manually instead of a committable suggestion. It +spends AI budget per run; operators can turn it off with `REVIEW_ADD_DOCS_ENABLED=false`. + +**`/improve`** — on demand, the bot runs an improvement pass over the whole PR and proposes +concrete changes the author can commit: clearer naming, dead or duplicated code, simpler +control flow, missing error handling, avoidable work in loops, and gaps in the tests covering +the change. It is deliberately separate from `/review`: `/review` looks for defects, +`/improve` proposes better code even when nothing is broken. Like a review, the pass is +token-budgeted rather than line-capped: the changed files are packed into batches that each fit +`REVIEW_MAX_INPUT_TOKENS`, one model call per batch, so a long diff only loses coverage once it +exceeds the whole budget. Each improvement whose quoted code anchors onto the diff is posted as +an inline committable `suggestion` block on the lines it replaces; the rest are listed as +copy-paste blocks in the run's summary comment, together with a partial-coverage note naming +any file the budget could not cover. Nothing is ever committed for you. It spends AI budget per +run — at most `REVIEW_MAX_AI_CALLS` model calls, the same ceiling as one review — and operators +can turn it off with `REVIEW_IMPROVE_ENABLED=false`. + +**`/generate-tests`** — on demand, the bot reads the diff and proposes unit tests for the +behavior the PR added or changed, in the test framework the project already uses. A proposed +test is usually a whole new file, which has no diff line for GitHub to anchor a committable +`suggestion` block to, so each one is posted as a code block headed by the path it belongs at +— ready to paste into a new file, or to merge into an existing test file. Nothing is committed +and no file is edited. Like a review, the pass is token-budgeted rather than line-capped: the +changed files are packed into batches that each fit `REVIEW_MAX_INPUT_TOKENS`, one model call +per batch, and the per-batch proposals are unioned by path — so "nothing here warrants a test" +is never a verdict on code that was too far down a long diff to be read. Any file the budget +could not cover is named in a partial-coverage note. It spends AI budget per run — at most +`REVIEW_MAX_AI_CALLS` model calls, the same ceiling as one review — and operators can turn it +off with `REVIEW_GENERATE_TESTS_ENABLED=false`. + + diff --git a/website/src/content/docs/0.6.2/comparison.md b/website/src/content/docs/0.6.2/comparison.md new file mode 100644 index 00000000..51e6c96e --- /dev/null +++ b/website/src/content/docs/0.6.2/comparison.md @@ -0,0 +1,89 @@ +--- +slug: 0.6.2/comparison +title: How it compares +description: An honest look at where ThrillhouseBot sits next to other AI code-review tools. +--- + + + + +A quick, honest look at where ThrillhouseBot sits next to other AI code-review +tools. Facts verified June 2026 from the sources linked below; vendor +capabilities change, so check the source if a detail matters to you. + +The table covers ThrillhouseBot, [CodeRabbit][cr], the open-source +[PR-Agent][pra], [GitHub Copilot code review][cop], the [PRSense][prs] CLI, and +[Kit][kit] (cased-kit). PR-Agent's commercial hosted sibling, Qodo Merge, is +left out because its self-hosting and model details sit behind enterprise sales +and could not be verified from public docs. PRSense and Kit are included because +they overlap on self-hosting and bring-your-own-model workflows; they differ in +how reviews are triggered and how cost visibility is surfaced. + +| | ThrillhouseBot | CodeRabbit | PR-Agent | Copilot review | PRSense | Kit | +|---|---|---|---|---|---|---| +| License | Apache-2.0 | Proprietary | Apache-2.0 [^pra-lic] | Proprietary | Apache-2.0 [^prs-lic] | MIT [^kit-lic] | +| Self-host | Yes | Enterprise only, 500+ seats [^cr-sh] | Yes | Models no; self-hosted runners yes [^cop-run] | Yes (CLI) [^prs] | Yes (CLI / Actions) [^kit] | +| Bring your own model | Any OpenAI-compatible endpoint | Yes: OpenAI, Azure OpenAI, Bedrock, Anthropic [^cr-llm] | Yes, via OpenAI-compatible / LiteLLM | No, GitHub-managed | Yes [^prs] | Yes [^kit] | +| Local models (Ollama) | Yes | No [^cr-llm] | Yes, with config caveats | No | Yes [^prs] | Yes [^kit] | +| Cost / token dashboard | Built-in web UI | No | No | GitHub billing only | Token counts in CLI only [^prs] | Per-review cost in CLI [^kit] | +| Footprint | ~50 MB native binary | SaaS or self-hosted container | Python app / container | SaaS plus Actions minutes [^cop-bill] | Node.js CLI [^prs] | Python app [^kit] | +| Cost model | Free; you pay your own API usage | Freemium | Free | Paid [^cop-bill] | Free; you pay your own API usage | Free; you pay your own API usage | + +## Where ThrillhouseBot fits + +Among the tools in the table above, ThrillhouseBot is the only one that combines +a GitHub App (webhook-driven reviews with inline PR comments), a built-in web +dashboard for cost and token analytics, token-budgeted map-reduce reviews for +large PRs, and a small native footprint, all under an OSI-approved license. If you want reviews to run on your own infrastructure +against a local Ollama model so that no code leaves your network, and you want +ongoing cost visibility in a dashboard rather than per-run CLI output, that is +the niche it targets. + +CodeRabbit and Copilot are more polished hosted products with broader reach. +ThrillhouseBot reviews GitHub pull requests only, with no GitLab or Bitbucket +support, and there is no managed hosting option: you run it yourself. + +PR-Agent, PRSense, and Kit are the closest in spirit for self-hosting with your +own model. PR-Agent is the most established open-source GitHub review bot in this +set. PRSense emphasizes grounded findings tied to the diff and ships as a CLI. +Kit adds rich repository context, local diff review, and per-run cost output in +the terminal or CI logs. None of those three ship a GitHub App plus web +dashboard in one package the way ThrillhouseBot does. + +Pick CodeRabbit or Copilot if you want a hosted product and don't need to bring +your own model. Pick PR-Agent, PRSense, or Kit if you prefer CLI or workflow +integration. Pick ThrillhouseBot if you want a GitHub App with automatic reviews, +a live dashboard, and a small native binary. + +[cr]: https://www.coderabbit.ai/ +[pra]: https://github.com/qodo-ai/pr-agent +[cop]: https://docs.github.com/en/copilot/concepts/agents/code-review +[prs]: https://prsense.org/ +[kit]: https://kit.cased.com/pr-reviewer/ + +[^pra-lic]: PR-Agent was moved to a community-owned GitHub organization and + relicensed under Apache-2.0 in April 2026. +[^cr-sh]: CodeRabbit docs: "The self-hosted option is only available for + CodeRabbit Enterprise customers with 500 user seats or more." + <https://docs.coderabbit.ai/self-hosted/github> +[^cr-llm]: Self-hosted CodeRabbit requires your own LLM credentials for OpenAI, + Azure OpenAI, AWS Bedrock, or Anthropic; its docs do not list local models + or Ollama. <https://docs.coderabbit.ai/self-hosted/github> +[^cop-run]: Copilot code review uses GitHub-managed models; you cannot supply + your own. It can run on self-hosted or larger GitHub-hosted runners. + <https://docs.github.com/en/copilot/concepts/agents/code-review> +[^cop-bill]: From June 1, 2026, Copilot code review is billed as AI credits for + token use plus GitHub Actions minutes for the review infrastructure. + <https://github.blog/changelog/2026-04-27-github-copilot-code-review-will-start-consuming-github-actions-minutes-on-june-1-2026/> +[^prs]: PRSense ships as the `@prsense/cli` npm package (Apache-2.0). Reviews + run locally or in CI; output includes token counts but there is no web + dashboard. Supports Ollama and cloud providers. + <https://prsense.org/> · <https://www.npmjs.com/package/@prsense/cli> +[^prs-lic]: `@prsense/cli` is published under Apache-2.0 on npm. +[^kit]: Kit (cased-kit) is a MIT-licensed Python CLI and library. Reviews can + run from the terminal, against local diffs, or from GitHub Actions; each run + reports LLM cost in stdout, not in a separate dashboard. + <https://kit.cased.com/pr-reviewer/> · <https://github.com/cased/kit> +[^kit-lic]: cased-kit is published under MIT on PyPI and GitHub. + + diff --git a/website/src/content/docs/0.6.2/configuration.md b/website/src/content/docs/0.6.2/configuration.md new file mode 100644 index 00000000..4d820ed8 --- /dev/null +++ b/website/src/content/docs/0.6.2/configuration.md @@ -0,0 +1,470 @@ +--- +slug: 0.6.2/configuration +title: Configuration +description: Every environment variable the bot reads, with defaults. +--- + + + +Configuration is read from environment variables (see `.env.example`). Short +names (`AI_*`, `REVIEW_*`, `WEBHOOK_*`, ...) are explicit aliases; every other +`thrillhousebot.*` key is settable through the standard Quarkus env-var mapping +— uppercase with `.`/`-` replaced by `_` (e.g. `thrillhousebot.review.ignored-files` +→ `THRILLHOUSEBOT_REVIEW_IGNORED_FILES`). The AI variables are the ones you +will change per provider: + +| Variable | Purpose | Default | +|---|---|---| +| `AI_API_KEY` | API key for the AI provider | _(required)_ | +| `AI_BASE_URL` | OpenAI-compatible base URL | `https://api.deepseek.com/v1` | +| `AI_MODEL` | Chat model name | `deepseek-chat` | +| `AI_PROVIDER` | Provider label for telemetry (`gen_ai.provider.name`); derived from `AI_BASE_URL` when unset | _(derived)_ | +| `AI_TIMEOUT` | Per-request timeout | `300s` | +| `AI_REASONING_ENABLED` | Send a reasoning hint to reasoning-capable models; when `false` no reasoning parameter is sent and the provider default applies | `false` | +| `AI_REASONING_EFFORT` | Effort sent while enabled: `none`/`low`/`medium`/`high`/`xhigh`/`max` (`none` explicitly asks the model not to reason; `xhigh`/`max` are the extended tiers newer reasoning models expose above `high`); reasoning tokens are billed as output tokens | `low` | +| `AI_REASONING_EFFORT_CONCISE` | Effort for the fixed-shape calls on the `concise` model (final summary, finding verifier, replies), which do **not** follow `AI_REASONING_EFFORT`: reasoning tokens count against `REVIEW_CONCISE_MAX_OUTPUT_TOKENS`, so a high effort there lets the verifier reason its whole allowance away and return an empty response. Same accepted values; unset means `low`, lowered to `AI_REASONING_EFFORT` when that is set below `low` | `low` | +| `GITHUB_APP_ID` | GitHub App ID | _(required)_ | +| `GITHUB_PRIVATE_KEY` | GitHub App private key (PEM) | _(required)_ | +| `GITHUB_WEBHOOK_SECRET` | Webhook HMAC secret | _(required)_ | +| `GITHUB_BOT_LOGINS` | Comma-separated bot account login(s) the bot skips to avoid replying to itself; override when deployed under a different App slug (`<app-slug>[bot]`) | `thrillhousebot[bot],thrillhouse-bot[bot]` | +| `GITHUB_WRITE_MIN_INTERVAL` | Duration spacing two content-creating GitHub calls (comments, review comments, thread replies, reviews), shared process-wide. GitHub secondary-rate-limits rapid content creation and answers `403`; pacing keeps the bot inside that envelope instead of discovering it by rejection — its published guidance is no more than one such request per second. `0` disables pacing | `1s` | +| `GITHUB_WRITE_MAX_WAIT` | Duration ceiling on how long one caller waits for its content-creation slot. Past it the call goes out unpaced and the bounded backoff handles a refusal, so a long queue never parks a finished command | `60s` | +| `WEBHOOK_DEDUP_TTL` | Webhook deduplication time-to-live for GitHub redeliveries | `24h` | +| `THRILLHOUSEBOT_REVIEW_MANUAL_TRIGGER_ALLOWED_LOGINS` | Comma-separated allowlist of logins permitted to run the slash commands without repo access; does not extend to the `@thrillhousebot resolved` directive, which always requires write access | _(empty)_ | +| `MANUAL_TRIGGER_AUTH_TIMEOUT` | Upper bound on the manual-trigger write-access check on the webhook ACK thread; fails closed (denies) if GitHub is slower | `5s` | +| `ACK_REACTION_TIMEOUT` | Upper bound on the 👀 command-ack reaction on the webhook ACK thread; the wait is abandoned (reaction may land late) if GitHub is slower | `3s` | +| `AUTO_REVIEW_MIN_INTERVAL` | Minimum interval between automatic reviews of the same PR — pushes within the window are skipped silently, even on a new head SHA (in-memory, per replica). A manual `/review` always bypasses; unset or `0` reviews every push | `0` (disabled) | +| `REVIEW_CI_GATING` | How strictly CI status factors into APPROVE: `strict` holds approval while required CI is pending, failing, or unreadable (fail-closed, safest); `warn` allows APPROVE but notes CI uncertainty in the summary/check; `off` skips CI entirely (findings-only). Prefer `strict` unless flaky CI or incomplete required-context resolution makes soft modes necessary | `strict` | +| `WEBHOOK_SKIP_DRAFTS` | Skip auto-review while a PR is a draft (reviewed once marked ready / on later pushes) | `false` | +| `WEBHOOK_REQUIRED_LABELS` | Comma-separated labels; only auto-review PRs carrying at least one (case-insensitive) | _(empty — no gate)_ | +| `WEBHOOK_EXCLUDED_LABELS` | Comma-separated labels; skip auto-review of PRs carrying any (wins over required) | _(empty)_ | +| `WEBHOOK_BASE_BRANCHES` | Comma-separated globs; only auto-review PRs whose base branch matches one (e.g. `main,release/*`). Globs are gitignore-style: `*` does **not** cross `/`, so use `**` to span slashes (`**` alone matches every branch) | _(empty — all branches)_ | +| `WEBHOOK_IGNORED_BASE_BRANCHES` | Comma-separated globs; skip auto-review of PRs whose base branch matches one (wins over allowlist; same `*`/`**` rule — match nested branches with `**`, e.g. `dependabot/**`) | _(empty)_ | +| `REVIEW_VERIFIER_ENABLED` | Second, skeptical AI pass that re-checks each finding against the diff before posting, dropping or downgrading what it can't confirm (see [AI call budget](#ai-call-budget)); fails open — a verifier error keeps the original findings | `true` | +| `REVIEW_DECLINE_RECHECK_ENABLED` | Re-check a maintainer's decline against the reviewed code before a prior finding is recorded "justified" (see [Re-checking declines](#re-checking-declines)); the finding stays open for one more round only when the reviewed diff plainly contradicts the stated reason. `false` makes a maintainer reply close the finding unconditionally | `true` | +| `REVIEW_BLOCKING_STRICTNESS` | When findings escalate to `REQUEST_CHANGES`: `balanced` (CRITICAL/HIGH + HIGH confidence), `strict` (any CRITICAL/HIGH), or `lenient` (CRITICAL + HIGH confidence only). See [Blocking strictness](#blocking-strictness) | `balanced` | +| `REVIEW_CONVERSATIONAL_REPLIES_ENABLED` | Answer `@thrillhousebot` mentions in PR threads (including finding replies) with an AI reply | `true` | +| `REVIEW_ADD_DOCS_ENABLED` | Allow the on-demand `/add-docs` command to generate docstrings as committable suggestions | `true` | +| `REVIEW_IMPROVE_ENABLED` | Allow the on-demand `/improve` command to run a whole-PR improvement pass and post committable suggestions | `true` | +| `REVIEW_GENERATE_TESTS_ENABLED` | Allow the on-demand `/generate-tests` command to propose unit tests for the changed code | `true` | +| `REVIEW_DIAGRAM_ENABLED` | Include an opt-in Mermaid control-flow diagram in the PR summary | `false` | +| `REVIEW_PATCH_COVERAGE_ENABLED` | Feed patch coverage into the review context: the added lines the repository's own coverage report records as never executed (see [Repository configuration](#repository-configuration)). Only takes effect for a repository that names its coverage artifact in `.github/thrillhousebot.yml` | `false` | +| `REVIEW_FOLLOW_UP_SUMMARY_ENABLED` | Post a short delta comment on follow-up reviews with the new-finding, resolved, and still-open counts. Only the first review posts the full summary; a follow-up pass with no delta (nothing new, nothing resolved) posts nothing | `false` | +| `REVIEW_LARGE_PR_NUDGE_ENABLED` | Add a note to the PR summary when a large PR's review opened **no inline finding** — it may be genuinely clean, or the pass may have been shallow — pointing at `/review` and `/improve`. Costs no extra AI call and never changes the verdict; a PR under both thresholds below is unaffected | `false` | +| `REVIEW_LARGE_PR_NUDGE_MIN_FILES` | Changed files at or above which the nudge applies (PR-level total, so ignored files still count). `0` switches this dimension off | `20` | +| `REVIEW_LARGE_PR_NUDGE_MIN_CHANGED_LINES` | Changed lines (additions + deletions) at or above which the nudge applies; either dimension triggers it on its own. `0` switches this dimension off — with both at `0` the nudge never fires | `1000` | +| `REVIEW_MAX_INPUT_TOKENS` | Per-call input-token budget for review, `/improve`, `/describe` and `/changelog` calls; large PRs are split into batches that each fit it. Bounded by the active model's input cap (see [Per-model AI settings](#per-model-ai-settings)). `0` disables token budgeting | `48000` | +| `REVIEW_OUTPUT_BUFFER_TOKENS` | Tokens reserved out of the input budget for the model's response | `8192` | +| `REVIEW_CONCISE_MAX_OUTPUT_TOKENS` | Response cap (`max_tokens`) for the fixed-shape/short AI calls — the final summary of a multi-call review, the finding verifier, and maintainer replies — which run on the `concise` named model so they don't share a cap sized for batch review output (see [Per-model AI settings](#per-model-ai-settings)). Hitting it surfaces as a truncation error naming this variable, never as a silently cut summary; set it empty to drop the cap and use the provider default | `8192` | +| `REVIEW_MAX_AI_CALLS` | Cap on AI calls per review (batch calls plus the final summary call), per `/describe` and `/changelog` run (batch calls plus one reduce call, spent only when the PR needed more than one batch), and per `/improve`, `/generate-tests` or `/add-docs` run (batch calls only — their results are merged locally); files that still don't fit are reported by name as omitted | `6` | +| `REVIEW_TOKEN_SAFETY_MARGIN` | Fraction of the input budget actually used, absorbing token-estimate error | `0.9` | +| `REVIEW_MAX_TOKENS_PER_REVIEW` | Ceiling on the tokens one review may consume across every AI call it makes — actual input+output as the provider reports them, counting retries and the final summary call, where `REVIEW_MAX_AI_CALLS` only counts planned calls. Once reached no further review call is made: remaining batches are disclosed by name as not reviewed (the verdict holds and the summary names the ceiling as the reason) and the summary degrades to a counts-only rendering that keeps the findings already paid for. `0` disables the ceiling. Review path only — the on-demand commands keep their own call cap | `0` | +| `REVIEW_MAX_DIFF_LINES` | Line cap on single-call diff renders (replies, base comparison, budgeting-disabled review). Token-budgeted reviews and the batched commands — `/improve`, `/describe`, `/changelog`, `/generate-tests`, `/add-docs` — ignore it (the planner owns coverage by tokens); `0` disables the cap | `5000` | +| `THRILLHOUSEBOT_REVIEW_MAX_REVIEW_COMMENTS` | Maximum inline comments posted per review; findings over the cap are surfaced in the summary instead of dropped | `50` | +| `THRILLHOUSEBOT_REVIEW_MAX_AI_RETRIES` | Attempts per failed AI call before the review errors out | `5` | +| `THRILLHOUSEBOT_REVIEW_AI_RETRY_BASE_DELAY_MS` | Base delay of the exponential retry backoff, in milliseconds | `2000` | +| `THRILLHOUSEBOT_REVIEW_AI_TIMEOUT_SECONDS` | Client-side wait per AI streaming attempt; keep it >= `AI_TIMEOUT` so timed-out attempts don't leave orphaned provider streams | `300` | +| `THRILLHOUSEBOT_REVIEW_INSTRUCTIONS_FILE` | Repo-relative path of the per-repo instructions file read on each review | `.github/thrillhousebot.md` | +| `THRILLHOUSEBOT_REVIEW_IGNORED_FILES` | Comma-separated gitignore-style globs excluded from review — lockfiles, generated code, build output. `*` does not cross `/`; use `**` to span directories. Replaces (not extends) the default list, so re-include the defaults you still want | `**/pom.xml,**/package-lock.json,**/*.lock,**/*.generated.*,**/target/**` | +| `THRILLHOUSEBOT_REVIEW_REPO_CONFIG_ENABLED` | Let each repository extend the ignore list with globs of its own, scope review rules to a path, and name its coverage-report artifact, from `.github/thrillhousebot.yml` (see [Repository configuration](#repository-configuration)). Both are additive; set `false` to make the deployment list and the global instructions the only ones that count | `true` | +| `REVIEW_LABELS_ENABLED` | Opt in to context-aware PR labels (see [PR labels](#pr-labels)) | `false` | +| `REVIEW_LABELS_APPLY` | When labels are enabled, add them to the PR instead of only suggesting them in a comment | `false` | +| `REVIEW_LABELS_ALLOW_CREATE` | Allow the bot to create suggested labels that don't exist yet | `false` | +| `REVIEW_LABELS_MAX` | Maximum labels applied or suggested per PR | `3` | +| `GITHUB_CLIENT_ID` / `GITHUB_CLIENT_SECRET` | OAuth credentials for dashboard login | _(required for dashboard)_ | +| `DASHBOARD_URL` | Public dashboard URL (OAuth callback base) | `http://localhost:8080` | +| `DATASOURCE_DB_KIND` | `h2` or `postgresql`. Quarkus fixes the datasource kind at build time, so this picks the driver when the app is built, not when a prebuilt image starts — released images are built for PostgreSQL and ignore an `h2` value | `h2` (dev), `postgresql` (`%prod`) | +| `HTTP_CONNECT_TIMEOUT` | Outbound HTTP connect timeout (GitHub API, OAuth) | `10s` | +| `HTTP_REQUEST_TIMEOUT` | Outbound HTTP request timeout (GitHub API, OAuth) | `10s` | +| `WEBSOCKET_KEEPALIVE_MS` | Dashboard WebSocket keepalive interval in ms; `0` or negative disables it (and stale replay-buffer eviction) | `25000` | + +### AI call budget + +A review that reports findings makes **two** model calls by default, not one: +the review call itself plus a verification call that re-sends the diff and the +candidate findings, so budget roughly **2× tokens** per flagged review. On +large PRs under token-aware budgeting this becomes N batch review calls + N +per-batch verification calls + one summary call. Set +`REVIEW_VERIFIER_ENABLED=false` to skip only the AI verifier — cheaper, at the +cost of more false positives; a deterministic hedging guard still runs, and a +verifier failure never blocks the review (it fails open, keeping the original +findings). + +The on-request commands are budgeted the same way. `/improve`, `/describe`, +`/changelog` and `/generate-tests` each split a large PR into batches under +`REVIEW_MAX_INPUT_TOKENS` and spend one call per batch, capped by +`REVIEW_MAX_AI_CALLS`. `/describe` and `/changelog` additionally reserve one call +of that cap for the step that reduces the per-batch results to a single +description or entry, and only spend it when the PR needed more than one batch — +so a single-batch PR still costs exactly one call. `/improve` and +`/generate-tests` reserve nothing, because their reductions are assembled locally +(a union of suggestions, and a union of proposed test files deduplicated by +path). When the cap is reached before every file has been batched, the uncovered +files are named in the partial-coverage note rather than dropped silently. + +### Re-checking declines + +When a maintainer replies to a finding to decline it, the follow-up analysis +records that finding as **justified** and the bot moves on. A dismissal is a +claim, though, not ground truth — a correct finding can be closed by an +incorrect rebuttal, and the rebuttal often names the very mechanism that makes +the bug real ("it only runs after the webhook is acked, so there's no race" — +on an executor that starts a thread per event). + +`REVIEW_DECLINE_RECHECK_ENABLED=true` (the default) therefore traces a decline's +stated reason against the code the review actually saw. When the reviewed diff +**plainly contradicts** that reason, the finding is kept **open for one more +round** with a note quoting both the claim and the contradicting line, instead +of being recorded justified. It is deliberately conservative: + +- Trusting the maintainer is the default. A rebuttal about house style, intent, + accepted risk, or priority — anything not refutable from the code — is + respected, as is any premise whose supporting code is not in the diff. +- **One push-back, then defer.** The re-check only fires while the thread carries + a single maintainer reply; replying again always ends it, so the bot can never + keep re-opening the same finding round after round. +- The re-opened finding is never re-posted as a new comment — it stays tracked in + *Previous Findings Status*, so nobody is asked to answer the same comment twice. +- The override holds approval (`APPROVE` → `COMMENT`) exactly like any other + unresolved previous finding; it never invents a new blocking finding. + +Set it to `false` to make a maintainer's reply final, unconditionally. + +### Clearing a finding with no thread + +Replying on a finding's review thread is the usual way to close it — but a +**LOW**-confidence finding at **MEDIUM** or **LOW** risk never opens one. It is +listed under **Things to double-check** in the summary instead, so there is no +thread to reply on, while follow-up reviews keep reporting it unresolved and +holding approval (`APPROVE` → `COMMENT`). To close one, comment on the PR +conversation: + +``` +@thrillhousebot resolved src/main/java/com/example/Widget.java:42 — Missing null check +``` + +Write it as plain text in the comment — pasted back inside a code fence or +backticks it reads as documentation and does nothing (see the quoting rule +below). + +Closing a finding wrongly is worse than leaving it open, so the match is strict +and **when the naming is ambiguous the finding stays held**. All of the +following must hold: + +- **The comment states `@thrillhousebot resolved`.** `resolve` (the thread + command) is not it, and a mention alone is not it — an ordinary question about + a finding is engagement, not a decision. Nor is the interrogative: + `@thrillhousebot resolved?` *asks* whether a finding was fixed, so it clears + nothing even when the rest of the comment names one. A question mark anywhere + later is fine — only one straight after the directive words makes it a question. +- **You name the finding by *both* its `path:line` locator and its own content** + — its title, or its description when it has no title. Each **Things to + double-check** row already prints both, the title first and then the locator in + backticks, so copying the row is the reliable way to get them right. Naming only + one of the two clears nothing. Locators match whole, so `Widget.java:42` never + closes a different finding at `Widget.java:4`. +- **You hold write access.** The comment's GitHub `author_association` must be + `OWNER`, `MEMBER` or `COLLABORATOR`; a fork-PR author's or a drive-by + commenter's comment is ignored. The bot's own comments are ignored too — its + summary reproduces every finding verbatim. +- **You *use* the directive rather than quote it.** `@thrillhousebot resolved` + counts as an instruction only when those words are plain text. Marking them up + — as `` `@thrillhousebot resolved` ``, inside a fenced block, or on a `>` + quoted line — reads as documentation, so a comment that *explains* the feature + (the fenced example above, or a colleague pasting one) clears nothing. The + **locator and title may still be in backticks**, which is how the summary + prints them; only the directive words themselves may not be. +- **Quoted blocks do not count at all.** Blockquote lines and fenced code are + dropped before anything is matched, so GitHub's *Quote reply* on the summary — + which reproduces every double-check row — names no finding either. + +One comment may name several findings; each is matched independently, and one it +does not name stays open. A finding with neither a title nor a description can +never be named this way, so it stays held — fix it, or reply on its thread if it +has one. The clearing is applied by the next review, which records the finding +**resolved**. + +The bot answers the directive as soon as it sees it, but only with what is +knowable that early: it has no review loaded and no findings to match against, so +it says what the next review will evaluate rather than reporting an outcome. The +one exception is a directive naming no `path:line` at all — that provably clears +nothing, so the reply says so outright and shows the shape to use. A locator that +is present but wrong still gets the general reply, and the review is what reports +the finding still unresolved. + +**Conversation read ceiling.** A review reads the PR conversation in pages of +100, up to 10 pages — 1000 comments — so one review can never turn a runaway +thread into hundreds of API calls. GitHub serves these comments oldest first and +offers no reverse order on that endpoint, so on a PR past the ceiling the +**newest** comments are the ones left unread, and a directive among them will not +clear anything that round. The bot logs a warning naming the ceiling whenever it +is reached, so this shows up as a log line rather than as the feature quietly +doing nothing; push a commit to re-review, or reply on the finding's thread if it +has one. + +### Blocking strictness + +By default (`REVIEW_BLOCKING_STRICTNESS=balanced`), only **CRITICAL** or **HIGH** +risk findings that the model reports at **HIGH** confidence escalate the PR +review to `REQUEST_CHANGES` (and fail the check run). Medium/low-confidence +severity findings still post as comments with a neutral check. + +| Mode | Blocks merge when | +|---|---| +| `balanced` (default) | CRITICAL or HIGH risk **and** HIGH confidence | +| `strict` | CRITICAL or HIGH risk, **any** confidence | +| `lenient` | CRITICAL risk **and** HIGH confidence only | + +**Security-team recommendation:** use `strict` so a CRITICAL/HIGH finding cannot +slip through as a comment just because confidence was demoted. Be aware that +under `strict`, the finding verifier's confidence demotions (hedged claims → +medium/low) no longer prevent a merge block — only risk reduction or dropping +the finding does. Stay on `balanced` if you want verifier demotions to keep +speculative severity findings non-blocking. + +This knob controls the **review verdict** only. Where findings are posted +(inline thread vs summary) is separate confidence gating tracked in +[#105](https://github.com/devops-thiago/ThrillhouseBot/issues/105); when that +lands, low-confidence findings can move out of the inline stream without +changing these blocking rules. + +The app validates configuration at startup and **fails fast** if a required value +(`GITHUB_APP_ID`, `GITHUB_PRIVATE_KEY`, `GITHUB_WEBHOOK_SECRET`, `AI_API_KEY`) is missing or — for +the private key — not a valid PEM RSA key, naming every offending variable in one message instead of +surfacing later on the first webhook or review. Dashboard OAuth (`GITHUB_CLIENT_ID` / +`GITHUB_CLIENT_SECRET`) is optional: leave both unset and the dashboard login is simply disabled. + +Cost tracking uses per-model pricing keyed by the model name, for example: + +```properties +thrillhousebot.ai.pricing.deepseek-chat.input-per-1k=0.00014 +thrillhousebot.ai.pricing.deepseek-chat.output-per-1k=0.00028 +``` + +If you switch to a different `AI_MODEL`, add a matching +`thrillhousebot.ai.pricing.<model>.*` pair so the dashboard can compute cost. +Without an entry the bot still records tokens, but warns once and flags sessions +as "no pricing" instead of showing `$0`. + +### Per-model AI settings + +Model-specific settings live under `thrillhousebot.ai.models.<model>.*`, keyed +by the model name (the `AI_MODEL` value) like the pricing map. Only the active +model's entry is read, so you can keep entries for every model you use and +switch `AI_MODEL` freely: + +```properties +# Input hard cap. The effective review budget is min(REVIEW_MAX_INPUT_TOKENS, +# cap); models without an entry get a 128000 cap. +thrillhousebot.ai.models.deepseek-chat.max-input-tokens=64000 +# The model's total context window. On a shared window the prompt and the +# completion are both charged to it, so boot fails when max-input-tokens + +# max-output-tokens do not fit inside it. Omit it and the ceiling isn't checked. +thrillhousebot.ai.models.deepseek-chat.context-tokens=128000 +# Per-model overrides of REVIEW_OUTPUT_BUFFER_TOKENS / REVIEW_TOKEN_SAFETY_MARGIN +thrillhousebot.ai.models.deepseek-chat.output-buffer-tokens=8192 +thrillhousebot.ai.models.deepseek-chat.token-safety-margin=0.9 +# Generation parameters, sent on every chat call when set +thrillhousebot.ai.models.deepseek-chat.temperature=0.2 +thrillhousebot.ai.models.deepseek-chat.top-p=0.95 +thrillhousebot.ai.models.deepseek-chat.max-output-tokens=8192 +thrillhousebot.ai.models.deepseek-chat.frequency-penalty=0.1 +thrillhousebot.ai.models.deepseek-chat.presence-penalty=0.1 +thrillhousebot.ai.models.deepseek-chat.seed=42 +# Set true only when the provider really bills the response outside the context +# window (1M in with 384K out on top, rather than 384K carved out of the 1M) — +# it switches off both the reservation and the context-tokens ceiling +thrillhousebot.ai.models.some-separate-budget-model.separate-output-budget=true +``` + +Notes: + +- **`max-input-tokens` is a cap, not the budget.** `REVIEW_MAX_INPUT_TOKENS` + stays the spend knob; the per-model value keeps it from overshooting the + model's real window. To use a large-context model beyond 128k, raise both. + Startup logs a warning whenever the cap lowers your configured budget. +- **`context-tokens` is the window itself**, and declaring it is what lets the + bot refuse an impossible request instead of paying for one. On a shared + window the provider charges the prompt *and* the completion to that one + context, so boot fails when `max-input-tokens + max-output-tokens` exceed it, + and again when your effective budget (`REVIEW_MAX_INPUT_TOKENS` clamped by the + cap) plus the largest response cap in play — `max-output-tokens` or + `REVIEW_CONCISE_MAX_OUTPUT_TOKENS` — exceeds it. Without it, an over-large + pair is only discovered when the provider rejects every call for length. + It is optional: a model that doesn't declare one is simply not checked. +- **Quote keys with `.` or `/`** (`thrillhousebot.ai.models."gpt-5.5".…`), the + same rule as the pricing map. Override via env — hyphen-only keys use underscores + (`THRILLHOUSEBOT_AI_MODELS_DEEPSEEK_V4_PRO_MAX_INPUT_TOKENS=1000000`); dotted keys use the + quoted-key form (`THRILLHOUSEBOT_AI_MODELS__GPT_5_5__MAX_INPUT_TOKENS=256000`). + `application.properties` ships empty stubs for known models so SmallRye can + disambiguate hyphenated keys — [Quarkus env mapping](https://quarkus.io/guides/config-reference#environment-variables). + For a model without a stub, add an empty + `thrillhousebot.ai.models."<model>".max-input-tokens=` line (external + `application.properties` or `-D`) alongside the env var. +- **`top_k` is not available** on the OpenAI-compatible wire; it becomes + relevant only with native provider integrations. +- **`max-output-tokens` no longer caps every call.** The final summary of a + multi-call review, the finding verifier, and maintainer replies run on a + second model binding — the `concise` named model + (`quarkus.langchain4j.openai.concise.*`) — that points at the same provider, + credentials, and model through the same `AI_*` variables and follows the + active model's temperature tuning, but carries its own response cap, + `REVIEW_CONCISE_MAX_OUTPUT_TOKENS` (default `8192`), and its own reasoning + effort, `AI_REASONING_EFFORT_CONCISE` (default `low`). Those responses are + fixed-shape or short, so they don't need — and shouldn't be licensed to + spend — a cap sized for batch review output. The review itself and the + command generators (`/describe`, `/changelog`, `/add-docs`, `/improve`, + `/generate-tests`), whose outputs scale with the diff, stay on the default + model and its `max-output-tokens`. +- **Reasoning effort is per lane.** Reasoning tokens are billed as output and + count against the response cap, so on the `concise` model a high effort can + consume the whole allowance and leave no content — the verifier then reports + that it kept its findings unverified. That tail is variable-length, not a + size threshold, so raising `REVIEW_CONCISE_MAX_OUTPUT_TOKENS` only shifts the + odds. Keep `AI_REASONING_EFFORT_CONCISE` low (the default) and spend the + effort budget on `AI_REASONING_EFFORT`, which drives the review itself. +- **`max-output-tokens` vs `output-buffer-tokens`**: `max-output-tokens` is the + hard response-length cap sent to the provider; `output-buffer-tokens` only + reserves input-budget headroom for the map-reduce budgeter. On a shared-window + model, keep the buffer at least as large as the output cap so a response the + model is allowed to produce always has reserved room — set both when capping + output. Boot fails if you don't. +- **`separate-output-budget`** (default `false`) says which contract the model is + on. Left off, prompt and completion share one window: the budgeter reserves + `output-buffer-tokens` out of the input budget, and the buffer must cover + `max-output-tokens`. Set it `true` for a model that publishes a response + allowance *on top of* its input window rather than inside it — then the + budgeter stops reserving (the response never draws on the diff budget), the + buffer no longer has to cover the cap, and the completion stops counting + against `context-tokens`. Getting it wrong is expensive in one direction and + unbootable in the other, so it is explicit rather than inferred: a + 384000-token cap on a 1M window silently costs ~40% of every call's diff + budget if the model is wrongly marked shared — while marking a genuinely + shared model separate turns off every guard and the provider rejects the + calls instead, which is how `deepseek-v4-flash` shipped its wrong pair. + Verify against the provider's own documented window before setting it. +- **`seed`** is a best-effort determinism hint (same seed + same parameters aims + for the same sampling) on providers that support it; unsupported providers + ignore it. For deterministic reviews, prefer a low `temperature` first. +- **Generation-parameter validation** happens at boot: temperature must be in + `[0, 2]`, `top-p` in `(0, 1]`, penalties in `[-2, 2]`, token counts positive — + a typo in any entry (even an inactive model's) fails startup with a message + naming the key. + + + + + +## Repository configuration + +The instructions file (`.github/thrillhousebot.md`) is prose for the model. +Structured settings live in a separate, optional `.github/thrillhousebot.yml` +(`.github/thrillhousebot.yaml` also works) — kept apart on purpose, because the +instructions fallback chain may land on a file owned by another tool, and its whole +content is fed to the model as untrusted prose: + +```yaml +review: + # Extra paths this repository never wants reviewed, on top of the deployment default. + ignored-files: + - "docs/generated/**" + - "**/*.snap" + - "testdata/**" + + # Review rules for one path only, on top of the prose in .github/thrillhousebot.md + # (which keeps applying everywhere). + path-instructions: + - path: "payments/**" + instructions: | + Money is handled in integer cents; flag any floating-point arithmetic. + Every state change must be idempotent under retry. + - path: "**/generated/**" + instructions: "Generated code: style and naming findings do not apply." + + # Name of the workflow artifact holding this repository's JaCoCo XML coverage report. + # Only read when the deployment sets REVIEW_PATCH_COVERAGE_ENABLED=true. + coverage-artifact: "coverage-report" +``` + +**Precedence: the effective ignore list is the union of both — global ∪ per-repo.** +A file is skipped if it matches *either* the deployment-wide +`thrillhousebot.review.ignored-files` list *or* a glob the repository declared. A +repository can therefore take more files out of review scope, but never put back a +file the deployment excludes, and a repository that ships no config file gets the +global list exactly as before. Globs use the same gitignore-style syntax as the +global key (`*` does not cross `/`; use `**` to span directories). + +**Precedence: the review rules for a file are the global instructions plus every +matching path scope.** The prose in `.github/thrillhousebot.md` (or whichever file the +fallback chain lands on) applies to every file exactly as before. On top of that, each +`path-instructions` scope whose glob matches a changed file contributes its rules *for +that file only* — the model is shown each scope alongside the files it governs, so +`payments/` strictness is never carried over to generated code. Scopes are additive and +may overlap: a file matching two scopes gets both, in declaration order, and where a +scope and the global instructions conflict, the scope wins for its own files. A scope +matching nothing in the pull request is not sent at all, and a file the ignore list +already excluded is never scoped — ignore rules run first. Path globs use the same +syntax and the same matcher as `ignored-files`. + +**Patch coverage: `coverage-artifact` names a report, and nothing is assumed without +it.** The bot never builds the pull request, so it cannot measure coverage itself. When +the deployment sets `REVIEW_PATCH_COVERAGE_ENABLED=true` *and* a repository names an +artifact here, each review looks for that artifact on a **completed workflow run for the +exact head commit** (GitHub's `head_sha` filter), downloads it, and intersects the +report's never-executed lines with the lines the diff adds. The reviewer is then shown a +short "uncovered changed lines" list and told two things: changed logic nothing exercises +is reportable, and a correctness claim about such a line must not be softened by the +usual "but a test in this diff covers it" check — no test runs that line. + +For this to work the workflow must upload the report, e.g. + +```yaml +- run: ./mvnw clean test jacoco:report +- uses: actions/upload-artifact@v7 + with: + name: coverage-report # must match review.coverage-artifact + path: target/site/jacoco/jacoco.xml +``` + +Only JaCoCo XML is understood today. **A repository that names no artifact — the common +case — contributes nothing, and its review is exactly what it was before.** The same is +true when the run uploaded nothing by that name, the artifact has expired, the download +fails, or the report is in another format: the section is omitted rather than guessed at. +Nothing about coverage is ever inferred from the diff, and a line's *absence* from the +list is explicitly not evidence that a test covers it. Files the ignore list already +excluded are never reported as under-tested. + +The file is read from the repository's default branch on each review and cached for +five minutes. Everything about it fails soft: a missing file, invalid YAML, an +unexpected shape, an uncompilable glob, or a malformed `path-instructions` entry is +logged and skipped, leaving the global ignore list and the global instructions in force +— it never fails a review. A repository may declare at most 25 scopes, each with at most +4000 characters of rules; the rest is dropped with a warning. Operators who do not want +repositories adjusting their own review scope or rules can turn the whole mechanism off +with `THRILLHOUSEBOT_REVIEW_REPO_CONFIG_ENABLED=false`. + + + + + +## PR labels + +ThrillhouseBot can suggest context-aware labels (area, change type, risk) drawn +from the diff. The feature is **off by default**; turn it on with +`REVIEW_LABELS_ENABLED=true`. + +When enabled, the model is shown the repository's existing labels and picks the +few that best describe the PR — it only ever chooses from labels that already +exist, so it respects whatever label scheme the repo already uses. What happens +next depends on `REVIEW_LABELS_APPLY`: + +- `false` (default): the suggestions are posted as a one-line comment on the + first review, leaving the decision to a maintainer. +- `true`: the labels are added to the PR automatically. + +Set `REVIEW_LABELS_ALLOW_CREATE=true` to let the bot create a suggested label +that doesn't exist yet (off by default, so it never invents labels), and +`REVIEW_LABELS_MAX` to cap how many labels it applies or suggests (default `3`). +Labelling is best-effort — a failure here never blocks or fails the review. + + diff --git a/website/src/content/docs/0.6.2/contributing.md b/website/src/content/docs/0.6.2/contributing.md new file mode 100644 index 00000000..c14152d5 --- /dev/null +++ b/website/src/content/docs/0.6.2/contributing.md @@ -0,0 +1,137 @@ +--- +slug: 0.6.2/contributing +title: Contributing +description: Development setup and the CI bar for contributions. +--- + + + + +This project follows the [Contributor Covenant](https://github.com/devops-thiago/ThrillhouseBot/blob/main/CODE_OF_CONDUCT.md). By taking +part you agree to uphold it. + +## Where to start + +- **[GitHub Discussions](https://github.com/devops-thiago/ThrillhouseBot/discussions)** — questions, setup help, and general conversation. Start with the pinned [welcome post](https://github.com/devops-thiago/ThrillhouseBot/discussions/1) if you are new. +- Issues labeled [`good first issue`](https://github.com/devops-thiago/ThrillhouseBot/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) are well-scoped and explained in detail. +- Issues labeled [`help wanted`](https://github.com/devops-thiago/ThrillhouseBot/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22) are larger but ready to be picked up. +- For bugs and feature requests, use the [issue templates](https://github.com/devops-thiago/ThrillhouseBot/issues/new/choose). For larger changes, open an issue first so we can discuss the approach before you invest time. + +## Development setup + +Prerequisites: **Java 25+**, **Node.js 22+** (dashboard only), **Docker** (native builds only). Maven is bundled via the wrapper. + +Follow the [README Quick Start](https://github.com/devops-thiago/ThrillhouseBot#quick-start) for cloning, credentials, dev mode, and webhook forwarding — it is the single source of truth for setup commands. Contributor-specific extras: + +```bash +cd frontend && npm install && npm run dev # dashboard with hot reload +``` + +## Before you open a PR + +Run the Java verification locally before opening a PR: + +```bash +./mvnw spotless:apply # google-java-format; CI runs spotless:check +./mvnw verify # tests + JaCoCo coverage gate + SpotBugs +``` + +If you changed the dashboard, also run `cd frontend && npm ci && npm run test && npm run build`. + +To exercise the UI without a backend, run `cd frontend && npm run dev:mock`. +CI additionally runs Dependency Review (required), SonarCloud, Trivy, and a Docker +build check on pull requests — see **Dual-gate merge policy** below. +To build a disposable test image for a branch or PR (JVM or native, without +touching `latest`), use the **Docker test image** workflow in Actions +(`.github/workflows/docker-test-image.yml`). + +The bar, enforced by CI: + +- **All tests pass** — and new code comes with tests. Test behavior, not implementation: prefer asserting observable outcomes over `assertDoesNotThrow`, and avoid reflection on private members (relax to package-private instead if a test genuinely needs the seam). +- **Coverage doesn't drop** — JaCoCo, Codecov patch coverage, and the SonarCloud quality gate all run on every PR. +- **SpotBugs clean** at `effort=Max`/`threshold=Low`. If you hit a false positive, prefer restructuring; document any exclusion in `config/spotbugs-exclude.xml` with a justification comment. +- **No new Sonar issues** — the quality gate requires an A reliability/security rating on new code. + +## Dual-gate merge policy + +ThrillhouseBot's LLM review and the repo's static CI gates are **complementary**, not substitutes. Merge only when **both** are green (or explicitly waived by a maintainer with a written reason on the PR). + +| Gate | What it catches | Required on this repo | +|------|-----------------|------------------------| +| **Static** | Dependency CVEs/license regressions ([Dependency Review](https://github.com/devops-thiago/ThrillhouseBot/blob/main/.github/workflows/dependency-review.yml)), filesystem CVEs (Trivy), format/tests/frontend, SpotBugs/Sonar/CodeQL | Yes — `format`, `test`, `frontend`, `trivy`, and `dependency-review` are required status checks on `main`/`develop` | +| **ThrillhouseBot** | Diff narrative, incomplete fixes, framework-context issues static tools miss | Soft gate — wait for the **ThrillhouseBot Review** check / posted review when the App is installed; do not merge solely because CI is green if the bot flagged open threads | + +Optional third signal: Cursor Bugbot (or similar) may run on PRs. It is **not** a merge requirement here — useful when present, never a replacement for the two gates above. + +This policy does **not** fold linters into the LLM prompt ([#34](https://github.com/devops-thiago/ThrillhouseBot/issues/34)); that work stays separate. The static gate is the CI safety net for supply-chain and compile-time classes of bug the bot can miss. + +### When the gates disagree + +Maintainers triage; do not "average" the signals away. + +| Static | ThrillhouseBot | What to do | +|--------|----------------|------------| +| Red | Green / quiet | **Fix or formally suppress the static finding.** LLM silence is not evidence the CVE is safe. Prefer a real version bump; if the advisory does not apply to this codebase, document the hold with the file the failing tool actually reads — Dependency Review: `allow-ghsas` or `.github/dependency-review-config.yml`; Trivy: `.trivyignore`; OpenSSF Scorecard / osv-scanner: `osv-scanner.toml` (Jackson GHSA pattern in [#308](https://github.com/devops-thiago/ThrillhouseBot/pull/308)). | +| Green | Red / open threads | **Address or refute the bot findings** in the PR discussion before merge. Static green does not clear logic/copy/incomplete-fix issues. | +| Red | Red | Clear the **static** gate first (often blocks merge already), then resolve bot threads. | +| Same area, conflicting severity | — | Prefer the **static** tool for dependency/CVE/license facts; prefer the **bot** for intent, call-site context, and "did the PR actually finish the fix?" Prefer neither blindly — leave a short maintainer note on the PR. | + +Waivers (rare): only a maintainer may merge with a red soft gate or a documented static suppression, and the PR must record who waived what and why. + +## Commit messages + +The project uses [Conventional Commits](https://www.conventionalcommits.org/): `feat:`, `fix:`, `perf:`, `test:`, `docs:`, `ci:`, `deps:`. One logical change per commit; explain the *why* in the body when it isn't obvious. + +## Architecture + +See the [architecture overview](https://devops-thiago.github.io/ThrillhouseBot/architecture/) +(source: [docs/ARCHITECTURE.md](https://github.com/devops-thiago/ThrillhouseBot/blob/main/docs/ARCHITECTURE.md)). Package flow: +`webhook/` → `review/` (`review/ai/`) → `github/` → `dashboard/` (`frontend/`). + +## Adding an AI provider + +There is no provider-specific code to write. The model is reached through +LangChain4j's OpenAI-compatible client, so a new provider is configuration: point +`AI_BASE_URL` and `AI_MODEL` at the endpoint, and add a matching +`thrillhousebot.ai.pricing.<model>.input-per-1k` / `.output-per-1k` pair in +`application.properties` if you want the dashboard to track its cost. The +[README provider table](https://github.com/devops-thiago/ThrillhouseBot#provider-support) lists the ones that are known +to work. + +## Prompt eval corpus + +Changes to the review or verifier prompts (`PrReviewPrompts`, `FindingVerifierPrompts`) +should be checked against the labeled regression corpus in +`src/test/resources/evalcorpus/` before they ship. Each case directory pins a real +dogfood outcome — a `case.json` spec plus `diff.txt` in the exact format the review +pipeline sends to the model: + +- **verifier** cases feed a candidate finding through the second-pass audit and assert + the expected verdict (`confirmed` / `downgraded` / `rejected`); +- **generator** cases run the first-pass review prompt over a diff and assert that a + finding matching the case's keywords is (`must-find`) or is not (`must-not-find`) + raised on the target file. + +The corpus schema is validated by `EvalCorpusTest` in every build. The live suite is +opt-in — it calls the configured AI provider: + +```bash +QUARKUS_LANGCHAIN4J_OPENAI_API_KEY=<key> ./mvnw test -Peval -Dtest=PromptEvalTest +``` + +Point `AI_BASE_URL` / `AI_MODEL` at the provider you want to evaluate (defaults apply +otherwise). Each case is sampled `-Deval.samples` times (default 3) and judged by +majority; `-Deval.tolerated` (default 0) allows a known-unfixed label — one tracked by +an open prompt-hardening issue — to fail without blocking unrelated prompt work. + +To add a case, create a new directory under `evalcorpus/` with `case.json` and +`diff.txt` (see an existing case for the shape). Source new cases from resolved PR +review threads: a refuted false positive becomes `expectedVerdicts: ["rejected"]`, a +confirmed true positive `["confirmed"]`, and note the provenance in `why`. + +## Reporting security issues + +Please **do not** open a public issue — see +[SECURITY.md](https://github.com/devops-thiago/ThrillhouseBot/blob/main/SECURITY.md). + + diff --git a/website/src/content/docs/0.6.2/feedback.md b/website/src/content/docs/0.6.2/feedback.md new file mode 100644 index 00000000..b320d71e --- /dev/null +++ b/website/src/content/docs/0.6.2/feedback.md @@ -0,0 +1,93 @@ +--- +slug: 0.6.2/feedback +title: Finding feedback +description: Data model, privacy, and retention for maintainer finding feedback signals. +--- + + + + +ThrillhouseBot records lightweight maintainer signals about review findings so a +future cross-review learnings store ([#38](https://github.com/devops-thiago/ThrillhouseBot/issues/38)) +has training data. This is the precursor shipped for +[#324](https://github.com/devops-thiago/ThrillhouseBot/issues/324); it does **not** +yet inject preferences into review prompts. + +## Why poll instead of a reaction webhook? + +GitHub Apps do not receive a `reaction` webhook event. The bot therefore lists +👍 (`+1`) and 👎 (`-1`) on finding comments via the +[Reactions REST API](https://docs.github.com/en/rest/reactions/reactions) when: + +1. A human **replies** on an inline review thread (`pull_request_review_comment` + with `in_reply_to_id`), or +2. A **follow-up review** already loaded inline comments — every bot finding-root + comment across prior rounds is scanned (capped, ordered by comment id), not + only findings from the immediately previous AI response. + +Capture is best-effort and never fails the webhook `200` or the review. + +## Signals + +| Signal | Source | Meaning | +|--------|--------|---------| +| `useful` | `reaction` (`+1`) | Maintainer marked the finding comment 👍 | +| `not_useful` | `reaction` (`-1`) | Maintainer marked the finding comment 👎 | +| `not_useful` | `reply_heuristic` | Reply body matched a conservative phrase (`not useful`, `false positive`, `noise`, 👎, `:-1:`) | + +Only comments that carry the hidden `<!-- thrillhousebot:finding=N -->` marker are +eligible. The bot's own reactions (e.g. 👀 command ack) are ignored. + +## Data model + +Table `finding_feedback` (Hibernate schema-update; no Flyway): + +| Column | Type | Notes | +|--------|------|-------| +| `id` | bigint | Panache / sequence PK | +| `repository` | string | `owner/repo` | +| `prNumber` | int | PR on that repository | +| `githubCommentId` | bigint | Finding root review-comment id | +| `findingIndex` | int (nullable) | 1-based index from the marker | +| `signal` | string | `useful` or `not_useful` | +| `source` | string | `reaction` or `reply_heuristic` | +| `reactorLogin` | string | GitHub login only (lower-cased) | +| `githubReactionId` | bigint (nullable) | Unique when present (idempotent re-poll) | +| `createdAt` | instant | Insert time | + +Unique constraints: + +- `githubReactionId` (when non-null) — reaction redeliveries / re-polls +- `(githubCommentId, reactorLogin, signal, source)` — one logical event per actor + +## Privacy + +Stored PII is limited to the **GitHub login** already present on webhook and API +payloads. No email, display name, IP, or reaction text beyond the fixed emoji +content codes (`+1` / `-1`) is persisted. Finding title/description are not +copied into this table. + +## Retention + +Rows are retained for the lifetime of the deployment database. There is no +automatic purge. Operators may `DELETE` rows or drop the table when +decommissioning an installation. Uninstalling the GitHub App does not currently +auto-delete feedback rows (same posture as `ReviewSession` history). + +## Aggregation API (ContextProvider seam) + +`FindingFeedbackService.summarize(repository)` and `summarizeAll()` return +per-repo `useful` / `not_useful` counts for a future `ContextProvider`. The +dashboard exposes the same aggregates at `GET /api/dashboard/feedback` (session +cookie required; optional `?repository=owner/repo`). + +## Notable classes + +- `FindingFeedback` / `FindingFeedbackRepository` / `FindingFeedbackService` +- `FindingFeedbackCaptureService` — poll + heuristics +- `GitHubReactionClient.listReviewCommentReactions` +- `WebhookController` — schedules capture on review-thread replies +- `ReviewOrchestrator` — capture pass on follow-up reviews + + + diff --git a/website/src/content/docs/0.6.2/getting-started.md b/website/src/content/docs/0.6.2/getting-started.md new file mode 100644 index 00000000..54cca7a4 --- /dev/null +++ b/website/src/content/docs/0.6.2/getting-started.md @@ -0,0 +1,125 @@ +--- +slug: 0.6.2/getting-started +title: Getting started +description: Create the GitHub App, configure the bot, and start it with Docker Compose. +--- + +Everything you need to go from zero to a bot reviewing your pull requests: +create the GitHub App, configure the bot, and start it with Docker Compose. + +## Prerequisites + +- [Docker & Docker Compose](https://docs.docker.com/compose/install/) +- An API key for any [OpenAI-compatible provider](/ThrillhouseBot/providers/) +- A public hostname where GitHub can reach the bot (for local development, + a [Smee.io](https://smee.io/) channel works) + +## 1. Create the GitHub App + +The bot authenticates as a GitHub App, so the app must exist before the bot +starts. The easiest path is the +**[hosted installer](/ThrillhouseBot/install.html)**: type the public hostname +where the bot will run and click **Create ThrillhouseBot GitHub App** — the +page builds the app manifest for you and sends it to GitHub. No local server +needed. + +After GitHub creates the app: + +1. Note the **App ID** (app settings → About). +2. Generate a **private key** (downloads a `.pem` file). +3. Set a **webhook secret**. +4. Copy the **Client ID** and **Client secret** from *Identifying and + authorizing users* (needed for dashboard login). +5. Install the app on your account or organization. + +Alternatively, grab the `code` query parameter GitHub appends to the +post-creation redirect and generate `.env` automatically from the manifest +conversion response: + +```bash +gh api --method POST /app-manifests/<code>/conversions \ + | java scripts/GenEnv.java --host <your-host> +``` + +:::note[Smee setups] +The hosted installer registers Smee webhooks at the channel root and the +OAuth callback at `http://localhost:8080/api/auth/callback`. If you use +`GenEnv.java`, don't pass the Smee URL as `--host` (that writes +`DASHBOARD_URL=https://<host>`); run it without `--host` and then set +`DASHBOARD_URL=http://localhost:8080` in the generated `.env` — dashboard +login only works when `DASHBOARD_URL` matches the registered callback. + +Webhooks land on the Smee channel, so also run the relay client to +forward them to the bot (keep it running alongside the bot): + +```bash +npx smee-client -u https://smee.io/YOUR_CHANNEL -t http://localhost:8080/api/webhook +``` +::: + +:::note[Manual registration instead] +Prefer to register the app by hand? Create it at +<https://github.com/settings/apps/new> with these settings: + +| Setting | Value | +|---|---| +| Webhook URL | `https://<your-host>/api/webhook` | +| Webhook Secret | Random string | +| Repository Permissions | Pull Requests: R/W, Checks: R/W, Contents: Read, Issues: R/W, Actions: Read, Commit Statuses: Read | +| Subscribe to Events | Pull Request, Issue comment, Pull request review comment | +| Identifying & authorizing users | Enabled (for dashboard login) | +| Callback URL | `https://<your-host>/api/auth/callback` | + +For a Smee channel, register the channel URL itself as the Webhook URL (no +`/api/webhook` suffix — Smee delivers only at the channel root) and +`http://localhost:8080/api/auth/callback` as the Callback URL, since OAuth +redirects happen in your browser against the local bot. +::: + +:::tip[Re-registering later] +Once the bot is running, `install.html` on the bot's own URL +(`https://<your-host>/install.html` behind a reverse proxy, or +`http://localhost:8080/install.html` when hitting it directly) builds the +manifest from the detected host automatically — handy for adding the app to +another account. Smee-based dev setups should re-register through the +[hosted installer](/ThrillhouseBot/install.html) instead: the bot's own page +registers the origin it's served from, without the Smee webhook-root handling. +::: + +## 2. Clone and configure + +```bash +git clone https://github.com/devops-thiago/ThrillhouseBot.git && cd ThrillhouseBot +cp .env.example .env +``` + +Edit `.env` with the credentials from step 1: + +| Variable | Value | +|---|---| +| `GITHUB_APP_ID` | From GitHub App settings → About | +| `GITHUB_PRIVATE_KEY` | The `.pem` downloaded when you generated a private key — on one line, newlines as `\n`, unquoted | +| `GITHUB_WEBHOOK_SECRET` | The webhook secret you set | +| `GITHUB_CLIENT_ID` | From app settings → Identifying and authorizing users | +| `GITHUB_CLIENT_SECRET` | From app settings → Identifying and authorizing users | +| `AI_API_KEY` | Your AI provider's API key | + +## 3. Start the bot + +```bash +docker compose up -d +``` + +The bot is running on `http://localhost:8080`. Point your reverse proxy at it +and you're done — the next pull request on an installed repository gets +reviewed automatically. + +## Next steps + +- Tune the bot with the + [configuration reference](/ThrillhouseBot/configuration/) — auto-review + triggers, labels, timeouts, and more. +- Drive it from a PR with the [comment commands](/ThrillhouseBot/commands/). +- Customize reviews per repository with a `.github/thrillhousebot.md` + instructions file (fallback chain: `.github/copilot-instructions.md` → + `CLAUDE.md` → `AGENTS.md` → `AGENT.md`). diff --git a/website/src/content/docs/0.6.2/index.md b/website/src/content/docs/0.6.2/index.md new file mode 100644 index 00000000..12694fca --- /dev/null +++ b/website/src/content/docs/0.6.2/index.md @@ -0,0 +1,78 @@ +--- +slug: 0.6.2 +title: ThrillhouseBot +description: Self-hosted AI pull-request reviewer — a GraalVM-native GitHub App built with Quarkus. +--- + +> **"Everything's coming up Thrillhouse!"** + +A self-hosted, GraalVM-native PR review bot, built as a GitHub App with Quarkus. +It reviews pull requests using any OpenAI-compatible chat API, so the review is +language-agnostic and you can pick the provider that suits you — including a +local Ollama model, so no code has to leave your network. + +![ThrillhouseBot approving a clean pull request](../../../assets/0.6.2/pr-approval.png) + +## Features + + + +- Reviews diffs for correctness, security, regressions, stale comments, and code quality +- Token-budgeted whole-PR review for large diffs — split into parallel map-reduce batches with omitted files named, not silently dropped +- Configurable auto-review triggers — skip drafts, gate on labels, or filter by base branch — plus an optional per-PR auto-review interval (`AUTO_REVIEW_MIN_INTERVAL`) when you want to cap spend on noisy PRs (off by default; use `/pause` to silence a PR) +- Inline code suggestions on review comments that you can apply with one click +- Every finding is tagged `critical`, `high`, `medium`, or `low` +- Follow-up reviews track whether earlier findings were addressed or justified +- Every finding can be closed by a maintainer: reply on its review thread, or — for one raised below the inline-posting bar, which has no thread — comment `@thrillhousebot resolved <path>:<line> — <title>` on the PR +- Maintainer 👍/👎 (and "not useful" replies) on finding comments are recorded for a future learnings pipeline — see [Finding feedback](https://devops-thiago.github.io/ThrillhouseBot/feedback/) +- Conversational replies: `@thrillhousebot` it in a PR thread or finding reply and the bot answers in context +- A summary comment on the first run, with a risk breakdown and a changed-files walkthrough +- Operable from the PR with comment commands — `/help`, `/review`, `/summary`, `/describe`, `/changelog`, `/add-docs`, `/improve`, `/generate-tests`, `/resolve`, `/pause`, `/resume` +- Live dashboard (Next.js) with a WebSocket activity feed, cost charts, and token tracking +- OpenTelemetry traces, token histograms, cost counters, and latency metrics +- Optional reasoning-effort dial and per-model generation/budget caps for OpenAI-compatible endpoints +- Reads per-repo instructions from `.github/thrillhousebot.md`, falling back to Copilot/Claude/Agents files +- Lets each repository add its own ignore globs in `.github/thrillhousebot.yml`, unioned with the deployment default, and scope extra review rules to a path glob +- Compiles ahead-of-time with GraalVM/Mandrel, so it starts fast and stays small + + + +## Where to go next + +- **[Getting started](/ThrillhouseBot/getting-started/)** — create the GitHub + App with the [hosted installer](/ThrillhouseBot/install.html) and run the bot + with Docker Compose. +- **[Commands](/ThrillhouseBot/commands/)** — drive the bot from a PR: + `/review`, `/describe`, `/changelog`, `/add-docs`, `/improve`, + `/generate-tests`, and more. +- **[Configuration](/ThrillhouseBot/configuration/)** — every environment + variable, with defaults. +- **[AI providers](/ThrillhouseBot/providers/)** — point the bot at the + OpenAI-compatible endpoint of your choice. +- **[Architecture](/ThrillhouseBot/architecture/)** — how a review flows + through the system. +- **[Finding feedback](/ThrillhouseBot/feedback/)** — maintainer 👍/👎 capture + for the learnings pipeline. +- **[How it compares](/ThrillhouseBot/comparison/)** — an honest look at where + ThrillhouseBot sits next to other AI code-review tools. +- **[Contributing](/ThrillhouseBot/contributing/)** — development setup and the + CI bar. + +## Dashboard + +The built-in dashboard (Next.js, served by the bot itself) shows summary cards, +a live activity feed that streams the model's output as a review runs, cost +charts by model, token breakdowns, and a paginated session history: + +![Dashboard Overview with summary cards, live model-output panel, and recent activity](../../../assets/0.6.2/live-streaming.png) + +## Community and license + +Questions and setup help belong in +[GitHub Discussions](https://github.com/devops-thiago/ThrillhouseBot/discussions); +bugs and feature requests in +[Issues](https://github.com/devops-thiago/ThrillhouseBot/issues/new/choose). + +Licensed under the +[Apache License 2.0](https://github.com/devops-thiago/ThrillhouseBot/blob/main/LICENSE) +(SPDX: `Apache-2.0`). diff --git a/website/src/content/docs/0.6.2/providers.md b/website/src/content/docs/0.6.2/providers.md new file mode 100644 index 00000000..3f828003 --- /dev/null +++ b/website/src/content/docs/0.6.2/providers.md @@ -0,0 +1,30 @@ +--- +slug: 0.6.2/providers +title: AI providers +description: Point the bot at any OpenAI-compatible chat endpoint. +--- + + + +ThrillhouseBot talks to any endpoint that implements the OpenAI chat-completions +API. Point `AI_BASE_URL` and `AI_MODEL` at your provider of choice: + +| Provider | `AI_BASE_URL` | Example `AI_MODEL` | +|---|---|---| +| DeepSeek | `https://api.deepseek.com/v1` | `deepseek-chat` | +| OpenRouter | `https://openrouter.ai/api/v1` | `openai/gpt-4o-mini` | +| Alibaba Cloud (Model Studio) | `https://dashscope-intl.aliyuncs.com/compatible-mode/v1` | `qwen-plus` | +| OpenAI | `https://api.openai.com/v1` | `gpt-4o-mini` | +| Ollama (local) | `http://localhost:11434/v1` | `llama3.2` | + +The default is DeepSeek, used only because it is inexpensive; nothing in the bot +is tied to it. + + + +There is no provider-specific code in the bot — a new provider is just +configuration. See +[Adding an AI provider](/ThrillhouseBot/architecture/#adding-an-ai-provider) +in the architecture notes, and add a `thrillhousebot.ai.pricing.<model>.*` pair +(see [Configuration](/ThrillhouseBot/configuration/)) if you want cost tracking +for the model. diff --git a/website/src/content/versions/0.6.2.json b/website/src/content/versions/0.6.2.json new file mode 100644 index 00000000..78217822 --- /dev/null +++ b/website/src/content/versions/0.6.2.json @@ -0,0 +1,40 @@ +{ + "sidebar": [ + { + "label": "Home", + "slug": "index" + }, + { + "label": "Getting started", + "slug": "getting-started" + }, + { + "label": "Commands", + "slug": "commands" + }, + { + "label": "Configuration", + "slug": "configuration" + }, + { + "label": "AI providers", + "slug": "providers" + }, + { + "label": "Architecture", + "slug": "architecture" + }, + { + "label": "Finding feedback", + "slug": "feedback" + }, + { + "label": "How it compares", + "slug": "comparison" + }, + { + "label": "Contributing", + "slug": "contributing" + } + ] +} diff --git a/website/versions.json b/website/versions.json index efd1d747..bc84be28 100644 --- a/website/versions.json +++ b/website/versions.json @@ -1,8 +1,12 @@ { "current": { - "label": "v0.6.2" + "label": "v0.6.3" }, "versions": [ + { + "slug": "0.6.2", + "label": "v0.6.2" + }, { "slug": "0.6.1", "label": "v0.6.1"