v1.28.0.0 feat: browse --headed/--proxy/--navigate + gstack/llms.txt + webdriver-only stealth#1363
Merged
v1.28.0.0 feat: browse --headed/--proxy/--navigate + gstack/llms.txt + webdriver-only stealth#1363
Conversation
Adds browse/src/socks-bridge.ts: a 127.0.0.1-only SOCKS5 listener that
accepts unauthenticated connections from Chromium and relays them through
an authenticated upstream proxy. Chromium does not prompt for SOCKS5 auth
at launch, so this bridge is the workaround for using auth-required
residential SOCKS5 upstreams.
- startSocksBridge({ upstream, port: 0 }) → ephemeral 127.0.0.1 listener
- testUpstream({ upstream, retries: 3, backoffMs: 500, budgetMs: 5000 })
pre-flight that connects to a known endpoint (default 1.1.1.1:443)
- Stream-error policy: kill affected client + upstream sockets on any
error mid-stream; no transport retries (a transport-layer retry can
corrupt browser traffic)
Adds browse/src/proxy-redact.ts: single source of truth for redacting
credentials in any logged proxy URL or upstream config. Every code path
that prints proxy config goes through this helper.
Adds the socks npm dep (~30KB) and 16 tests covering: 127.0.0.1-only
bind, byte-for-byte round trip through the bridge, auth rejection,
mid-stream upstream drop kills client conn, listener teardown,
testUpstream success + retry-exhaust paths, redaction of every
credential shape.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the global --proxy <url> and --headed flags to the browse CLI.
Resolves cred policy and routes the daemon launch through the SOCKS5
bridge (or pass-through for HTTP/HTTPS) before chromium.launch().
CLI (cli.ts):
- extractGlobalFlags() strips --proxy/--headed from argv, parses URL via
Node URL class, validates D9 cred-mixing (env BROWSE_PROXY_USER/PASS
+ URL creds → exit 1 with hint), composes canonical proxy URL with
resolved creds, computes a stable configHash for daemon-mismatch
- ensureServer() now reads existing daemon's configHash from state file
and refuses (exit 1 with disconnect hint) if --proxy/--headed mismatch
the existing daemon. No silent restart that would drop tab state.
- All proxy-related stderr lines go through redactProxyUrl
proxy-config.ts (new):
- parseProxyConfig() — URL parser + D9 cred-mixing detector + scheme allowlist
- computeConfigHash() — stable hash of (proxy URL minus creds + headed flag)
- toUpstreamConfig() — map ParsedProxyConfig → socks-bridge.UpstreamConfig
Server (server.ts):
- Reads BROWSE_PROXY_URL at startup; for SOCKS5+auth, runs testUpstream
pre-flight (5s budget, 3 retries, 500ms backoff) and exits 1 on failure
with redacted error
- Spawns startSocksBridge() on 127.0.0.1:<ephemeral> and points
Chromium at it via socks5://127.0.0.1:<port>
- HTTP/HTTPS or unauth SOCKS5 → pass-through to chromium.launch
proxy.server (with username/password if present)
- State file gains optional configHash for daemon-mismatch check
- Bridge tears down via process.on('exit')
Browser manager (browser-manager.ts):
- New setProxyConfig({ server, username, password }) called by server.ts
before launch
- chromium.launch() and both launchPersistentContext sites pass the
proxy config through when set
Tests: 22 new across proxy-config (parse + cred-mixing + hash stability)
and extractGlobalFlags (flag stripping + cred-mixing rejection + cred
rotation hash stability + redaction).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds browse/src/xvfb.ts: a Linux-only Xvfb auto-spawn module for
running headed Chromium in containers without DISPLAY. The module
walks a display range to pick a free one (never hardcodes :99) and
validates orphan PIDs by BOTH /proc/<pid>/cmdline matching 'Xvfb' AND
start-time matching the recorded value before sending any signal.
Defends against PID reuse — refuses to kill anything that doesn't
match both checks.
- shouldSpawnXvfb(env, platform) — pure decision: skip on macOS/Windows,
on Linux skip when DISPLAY or WAYLAND_DISPLAY is set (codex F2)
- pickFreeDisplay(99..120) — probes via xdpyinfo
- spawnXvfb(display) — returns { pid, startTime, display } handle
- isOurXvfb(pid, startTime) — both-checks validator
- cleanupXvfb(state) — best-effort, validates ownership before SIGTERM
Wired into server.ts startup: when shouldSpawnXvfb says yes, picks a
free display, spawns Xvfb, sets DISPLAY for chromium.launchHeaded, and
records xvfbPid/xvfbStartTime/xvfbDisplay in the state file. Cleanup
runs on process.on('exit'). The CLI's disconnect path also runs
cleanupXvfb() in the force-cleanup branch when the server is dead.
Disconnect now applies to any non-default daemon (headed mode OR
configHash-tagged daemon — i.e. one started with --proxy/--headed),
not just headed mode.
Adds xvfb + x11-utils to .github/docker/Dockerfile.ci so CI exercises
the Linux container --headed path on every run. Without it the most
common production path would go untested.
Tests: 17 new across decision logic, PID validation defenses
(cmdline mismatch, start-time mismatch), no-op safety on bad inputs,
and a Linux+Xvfb-installed gate for the spawn → validate → cleanup
round trip. Tests skip on macOS/Windows automatically.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
D7 (codex narrowing): mask navigator.webdriver only via addInitScript. The wintermute approach (fake plugins=[1..5], fake languages=['en-US', 'en'], stub window.chrome) is intentionally NOT applied — modern fingerprinters check consistency between plugins.length, languages, userAgent, and platform, and synthesizing fixed values can flag MORE bot-like, not less. The honest minimum is webdriver, which Chromium exposes as a known automation tell. Adds browse/src/stealth.ts: single source of truth for the stealth init script and launch args. Both browser-manager.launch() (headless) and launchHeaded() (persistent context with extension) call applyStealth(context) and pass STEALTH_LAUNCH_ARGS into chromium.launch. The pre-existing launchHeaded stealth that did fake plugins/languages is removed for the same reason. The cdc_/__webdriver runtime cleanup and Permissions API patch are kept — they remove automation-injected artifacts, not synthesize fake natural-browser values. Adds bridge-chromium-e2e.test.ts (codex F3): the test that proves the FEATURE works. Real Chromium with proxy.server = 'socks5://127.0.0.1: <bridgePort>' navigates to a local HTTP fixture; the auth upstream's connect counter and the HTTP fixture's hit counter both increment, proving traffic actually traversed bridge → auth-upstream → destination. Without this test, we could ship a working byte-relay and a broken Chromium integration and never know. Adds bridge-port-restart.test.ts (codex F1, reframed): old test assumed two daemons coexist, which contradicts D2 single-daemon model. Reframed as restart-then-restart, asserting fresh ephemeral ports (never the hardcoded 1090) on each spin-up. Adds stealth-webdriver.test.ts: navigator.webdriver=false in both fresh contexts and persistent contexts; navigator.plugins/languages are NOT replaced with the wintermute fake list (D7 verification). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… agents Adds scripts/gen-llms-txt.ts: produces gstack/llms.txt at repo root, indexing every skill (47), every browse command (75), and design commands when the design CLI is present. Per the llmstxt.org convention, agents can read one file to learn what gstack offers instead of crawling 47 SKILL.md files. Sources: - skill SKILL.md.tmpl frontmatter (name + description block scalar) - browse/src/commands.ts COMMAND_DESCRIPTIONS (sorted by category) - design/src/commands.ts COMMAND_DESCRIPTIONS if present (best-effort) Wired into scripts/gen-skill-docs.ts as a post-step so it regenerates on every `bun run gen:skill-docs` (the same script that re-emits all SKILL.md files). Failures are non-fatal warnings, not build breaks — the generator never blocks SKILL.md regen. Strict mode (--strict, also used by tests) throws when a skill is missing name or description in its frontmatter, catching missing metadata before it ships. Tests: shape (top-level sections, sort order, single-line summary discipline), every-skill-and-command-appears, strict-mode rejection of incomplete frontmatter, and freshness check that the committed gstack/llms.txt matches what the generator produces now. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the --navigate strategy from community PR #1355 (originally from @garrytan-agents). When set, download navigates to the URL with waitUntil:'commit' and captures the resulting browser download via page.waitForEvent('download'), then saves via download.saveAs(). Handles URLs that trigger files via Content-Disposition headers, multi-hop CDN redirects requiring browser cookies, or anti-bot CDN chains where page.request.fetch() can't follow the auth/redirect chain. Defaults still use the existing direct-fetch strategy. --navigate is opt-in. Goes through the same validateNavigationUrl SSRF gate as goto, so download --navigate cannot reach IPv4 metadata endpoints (AWS IMDSv1, GCP/Azure equivalents) or arbitrary internal hosts. Inferred content type from suggested filename for common extensions (epub, pdf, zip, gz, mp3/mp4, jpg/jpeg/png, txt, html, json) — falls back to application/octet-stream. Same 200MB cap as Strategy 1. Frames the use case generically (anti-bot CDN, Content-Disposition, redirect chains) rather than naming any specific site, per project voice rules. Co-Authored-By: @garrytan-agents Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
VERSION 1.27.1.0 → 1.28.0.0 (MINOR — substantial new capability: five new flags/features, ~600 LOC added, new socks dep, multiple new modules). browse/SKILL.md.tmpl: new "Headed Mode + Proxy + Anti-Bot Sites" section between User Handoff and Snapshot Flags. Documents --headed (auto-Xvfb on Linux), --proxy (with embedded SOCKS5 bridge for auth), download --navigate, the cred-mixing policy, daemon-discipline (refuse-on-mismatch), the narrowed webdriver-only stealth, container support caveats, and the fail-fast/no-retry failure modes. CHANGELOG entry follows the release-summary format from CLAUDE.md: two-line headline, lead paragraph, "The numbers that matter" table tied to specific test files that prove each capability, "What this means for AI agents" closing tied to a real workflow shift, then itemized Added/Changed/Fixed/For-contributors sections. Browse SKILL.md regenerated via bun run gen:skill-docs. gstack/llms.txt regenerated automatically from the same pipeline. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds two integration tests that exercise the full process boundary, not just the module-level wiring. daemon-mismatch-refuse.test.ts (D2): - Stubs a healthy state file with a fake configHash and a fake /health HTTP server, runs the actual cli.ts binary with a mismatching --proxy, asserts exit 1 + 'different config' / 'browse disconnect' hint in stderr. - Same shape with the plain-daemon-meets---headed case. - Positive case: matching configHash → CLI does NOT emit the mismatch hint (regardless of whether the actual command succeeds). server-proxy-fail-fast.test.ts: - Starts the rejecting SOCKS5 upstream, spawns server.ts with BROWSE_PROXY_URL pointing at it, BROWSE_HEADLESS_SKIP=1 to skip Chromium launch. - Asserts exit 1, 'FAIL upstream' in stderr (testUpstream pre-flight ran), no raw credential leakage in any output (redaction works on the failure path), and exit within 30s upper bound. Both tests use the existing spawn-bun-cli pattern from commands.test.ts so they run on the same CI infrastructure as the rest of the bun test suite. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two regressions caught by the full test suite after the v1.28.0.0
landing pass:
1) package.json version mismatch — VERSION was bumped to 1.28.0.0
but package.json still pinned to 1.27.1.0.
test/gen-skill-docs.test.ts asserts they match.
2) Top-level await in scripts/gen-llms-txt.ts (CLI entry block) and
scripts/gen-skill-docs.ts (post-step) made gen-skill-docs an
async module. test/gen-skill-docs.test.ts uses require() to pull
extractVoiceTriggers/processVoiceTriggers from gen-skill-docs,
which Bun rejects on async modules with:
"TypeError: require() async module ... unsupported.
use 'await import()' instead."
Fix: wrap the await blocks in void IIFEs so the modules remain sync
from a require() perspective.
After fix: all 379 gen-skill-docs tests pass, all 77 new feature
tests pass (3 skipped on macOS — Linux+Xvfb gates).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codex outside-voice review caught five real production-failure modes in
the v1.28.0.0 proxy/headed lifecycle. Fixed:
1) `browse disconnect` skip-graceful for proxy-only daemons
(browse/src/cli.ts). The graceful /command POST went out with stray
`domains,` shorthand and (even fixed) the server's disconnect handler
only tears down headed mode — proxy-only daemons returned 200 "Not
in headed mode" while leaving the bridge running. Now disconnect
short-circuits to force-cleanup for non-headed daemons, which kicks
process.on('exit') in server.ts to close the bridge + Xvfb.
2) sendCommand crash retry preserves --proxy / --headed
(browse/src/cli.ts). The ECONNRESET retry path called startServer()
with no extraEnv, silently dropping the proxied flags. A daemon that
died mid-command would silently restart in default direct/headless
mode and bypass the SOCKS bridge. Now reapplies BROWSE_PROXY_URL,
BROWSE_HEADED, and BROWSE_CONFIG_HASH from the resolved global flags.
3) `connect` honors --proxy (browse/src/cli.ts). The headed-mode
`connect` command built its own serverEnv that didn't include
BROWSE_PROXY_URL, so `browse --proxy <url> connect` launched headed
Chromium without the proxy. Now threads proxyUrl + configHash into
the connect serverEnv.
4) SOCKS5 bridge handles fragmented TCP frames
(browse/src/socks-bridge.ts). Previously used once('data') and
parsed each chunk as a complete SOCKS5 frame — TCP doesn't preserve
message boundaries and split greetings/CONNECT requests caused
intermittent handshake failures. Replaced with a single state
machine that buffers chunks and uses size predicates on the SOCKS5
header to know when a complete frame has arrived. Pauses the client
socket during upstream connect and replays any remainder bytes
into the upstream on success.
5) Xvfb cleanup-then-state-delete ordering
(browse/src/server.ts). emergencyCleanup() previously deleted the
state file BEFORE any Xvfb cleanup could read it, orphaning Xvfb
on uncaughtException / unhandledRejection. Now reads the state
file first, calls cleanupXvfb() (which validates cmdline +
start-time before kill), then deletes the state file.
Adds a regression test for #4: writes the SOCKS5 greeting + CONNECT
one byte at a time with 5ms ticks, asserts a clean round trip after
the fragmented handshake.
Codex's sixth finding (bridge advertises NO_AUTH on 127.0.0.1, so any
co-located process can use the authenticated upstream) is documented
as a known limitation — gstack's threat model assumes single-user
hosts. Adding bridge-side auth is a separate change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
BROWSER.md picks up a "Headed mode + proxy + browser-native downloads (v1.28.0.0)" subsection inside Real-browser mode plus the new source-map entries (socks-bridge.ts, proxy-config.ts, proxy-redact.ts, xvfb.ts, stealth.ts). TODOS.md anti-bot-stealth item updated to reflect the v1.28 narrowing — the "fake plugins" line is no longer accurate. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CI evals all failed on PR #1363 with: error: Could not resolve: "smart-buffer". Maybe you need to "bun install"? error: Could not resolve: "ip-address". Maybe you need to "bun install"? at /opt/node_modules_cache/socks/build/client/socksclient.js:15 The cached node_modules layer in the pre-baked Docker image had `socks` (the new dep) but was missing its transitive deps (smart-buffer, ip-address). The image build copied only package.json into the build context — without bun.lock, `bun install` resolved a different tree than local `bun install` did, dropping required transitive deps. Reproduces locally as 229 packages (correct) when bun.lock is present or absent. Why CI diverged isn't fully understood — possibly Docker layer cache reuse across image rebuilds — but the deterministic fix is to include the lockfile in the image build context and use `--frozen-lockfile`, matching what every CI doc recommends. Changes: - .github/docker/Dockerfile.ci: COPY bun.lock alongside package.json, switch `bun install` → `bun install --frozen-lockfile` so any future lockfile drift fails loudly during image build instead of producing a partially-installed cache that breaks downstream eval jobs. - .github/workflows/evals.yml: include bun.lock in the image-tag hash so adding/removing a dep invalidates the image, AND copy bun.lock into the docker context alongside package.json. - .github/workflows/evals-periodic.yml: same updates. - .github/workflows/ci-image.yml: rebuild trigger now fires on bun.lock changes too; build context includes bun.lock. Image hash changes → fresh image gets built on next CI run → install matches the lockfile exactly → no missing transitive deps. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After the bun.lock fix landed, the eval matrix STILL failed identically: Could not resolve: "smart-buffer" / "ip-address" at /opt/node_modules_cache/socks/build/client/socksclient.js But the hash-tagged image actually contains smart-buffer + ip-address + socks all flat in /opt/node_modules_cache (verified by pulling and inspecting the image). 207 packages, all present. Root cause: the workflow used `ln -s /opt/node_modules_cache node_modules` to restore deps. Bun build (and Node module resolution generally) walks a file's realpath to find sibling deps. From the symlinked /workspace/node_modules/socks/build/client/socksclient.js, realpath resolves to /opt/node_modules_cache/socks/build/client/socksclient.js, and walking up to find a node_modules/smart-buffer dir fails — there's no `node_modules` segment in the realpath. Switch `ln -s` → `cp -al` (hardlink-copy). Each file in the cache becomes a hardlink at /workspace/node_modules/<pkg>, sharing inodes (no data copy). Realpath of /workspace/node_modules/socks/.../socksclient.js stays inside /workspace/node_modules, so sibling deps resolve correctly. Speed is comparable to symlink — `cp -al` on ~200 packages on tmpfs is sub-second. Same caching story preserved. Both evals.yml and evals-periodic.yml updated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…filesystems
The hardlink-copy fix landed and immediately broke with:
cp: cannot create hard link 'node_modules/<file>' to
'/opt/node_modules_cache/<file>': Invalid cross-device link
GitHub Actions runners mount the workspace volume at /workspace
(overlay-fs layered onto the runner image), and /opt is the runner
image's own filesystem. Cross-filesystem hardlinks aren't supported.
Switch `cp -al` → `cp -r`. Cost: ~5s for ~200 packages of small JS
files vs ~0s for the broken symlink. Still cheaper than the ~15s
`bun install` fallback. Realpath of /workspace/node_modules/<pkg>/...
stays inside /workspace, so bun build's sibling-dep resolution works.
Both evals.yml and evals-periodic.yml updated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
E2E Evals: ✅ PASS30/30 tests passed | $3.72 total cost | 12 parallel runners
12x ubicloud-standard-2 (Docker: pre-baked toolchain + deps) | wall clock ≈ slowest suite |
1 task
gonnabe88
pushed a commit
to gonnabe88/gstack
that referenced
this pull request
May 9, 2026
…+ webdriver-only stealth (garrytan#1363) * feat(browse): SOCKS5 bridge with auth + cred redaction helper Adds browse/src/socks-bridge.ts: a 127.0.0.1-only SOCKS5 listener that accepts unauthenticated connections from Chromium and relays them through an authenticated upstream proxy. Chromium does not prompt for SOCKS5 auth at launch, so this bridge is the workaround for using auth-required residential SOCKS5 upstreams. - startSocksBridge({ upstream, port: 0 }) → ephemeral 127.0.0.1 listener - testUpstream({ upstream, retries: 3, backoffMs: 500, budgetMs: 5000 }) pre-flight that connects to a known endpoint (default 1.1.1.1:443) - Stream-error policy: kill affected client + upstream sockets on any error mid-stream; no transport retries (a transport-layer retry can corrupt browser traffic) Adds browse/src/proxy-redact.ts: single source of truth for redacting credentials in any logged proxy URL or upstream config. Every code path that prints proxy config goes through this helper. Adds the socks npm dep (~30KB) and 16 tests covering: 127.0.0.1-only bind, byte-for-byte round trip through the bridge, auth rejection, mid-stream upstream drop kills client conn, listener teardown, testUpstream success + retry-exhaust paths, redaction of every credential shape. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(browse): --proxy and --headed flags wire bridge into daemon Adds the global --proxy <url> and --headed flags to the browse CLI. Resolves cred policy and routes the daemon launch through the SOCKS5 bridge (or pass-through for HTTP/HTTPS) before chromium.launch(). CLI (cli.ts): - extractGlobalFlags() strips --proxy/--headed from argv, parses URL via Node URL class, validates D9 cred-mixing (env BROWSE_PROXY_USER/PASS + URL creds → exit 1 with hint), composes canonical proxy URL with resolved creds, computes a stable configHash for daemon-mismatch - ensureServer() now reads existing daemon's configHash from state file and refuses (exit 1 with disconnect hint) if --proxy/--headed mismatch the existing daemon. No silent restart that would drop tab state. - All proxy-related stderr lines go through redactProxyUrl proxy-config.ts (new): - parseProxyConfig() — URL parser + D9 cred-mixing detector + scheme allowlist - computeConfigHash() — stable hash of (proxy URL minus creds + headed flag) - toUpstreamConfig() — map ParsedProxyConfig → socks-bridge.UpstreamConfig Server (server.ts): - Reads BROWSE_PROXY_URL at startup; for SOCKS5+auth, runs testUpstream pre-flight (5s budget, 3 retries, 500ms backoff) and exits 1 on failure with redacted error - Spawns startSocksBridge() on 127.0.0.1:<ephemeral> and points Chromium at it via socks5://127.0.0.1:<port> - HTTP/HTTPS or unauth SOCKS5 → pass-through to chromium.launch proxy.server (with username/password if present) - State file gains optional configHash for daemon-mismatch check - Bridge tears down via process.on('exit') Browser manager (browser-manager.ts): - New setProxyConfig({ server, username, password }) called by server.ts before launch - chromium.launch() and both launchPersistentContext sites pass the proxy config through when set Tests: 22 new across proxy-config (parse + cred-mixing + hash stability) and extractGlobalFlags (flag stripping + cred-mixing rejection + cred rotation hash stability + redaction). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(browse): Xvfb auto-spawn with PID + start-time validation Adds browse/src/xvfb.ts: a Linux-only Xvfb auto-spawn module for running headed Chromium in containers without DISPLAY. The module walks a display range to pick a free one (never hardcodes :99) and validates orphan PIDs by BOTH /proc/<pid>/cmdline matching 'Xvfb' AND start-time matching the recorded value before sending any signal. Defends against PID reuse — refuses to kill anything that doesn't match both checks. - shouldSpawnXvfb(env, platform) — pure decision: skip on macOS/Windows, on Linux skip when DISPLAY or WAYLAND_DISPLAY is set (codex F2) - pickFreeDisplay(99..120) — probes via xdpyinfo - spawnXvfb(display) — returns { pid, startTime, display } handle - isOurXvfb(pid, startTime) — both-checks validator - cleanupXvfb(state) — best-effort, validates ownership before SIGTERM Wired into server.ts startup: when shouldSpawnXvfb says yes, picks a free display, spawns Xvfb, sets DISPLAY for chromium.launchHeaded, and records xvfbPid/xvfbStartTime/xvfbDisplay in the state file. Cleanup runs on process.on('exit'). The CLI's disconnect path also runs cleanupXvfb() in the force-cleanup branch when the server is dead. Disconnect now applies to any non-default daemon (headed mode OR configHash-tagged daemon — i.e. one started with --proxy/--headed), not just headed mode. Adds xvfb + x11-utils to .github/docker/Dockerfile.ci so CI exercises the Linux container --headed path on every run. Without it the most common production path would go untested. Tests: 17 new across decision logic, PID validation defenses (cmdline mismatch, start-time mismatch), no-op safety on bad inputs, and a Linux+Xvfb-installed gate for the spawn → validate → cleanup round trip. Tests skip on macOS/Windows automatically. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(browse): webdriver-mask stealth + Chromium-through-bridge e2e D7 (codex narrowing): mask navigator.webdriver only via addInitScript. The wintermute approach (fake plugins=[1..5], fake languages=['en-US', 'en'], stub window.chrome) is intentionally NOT applied — modern fingerprinters check consistency between plugins.length, languages, userAgent, and platform, and synthesizing fixed values can flag MORE bot-like, not less. The honest minimum is webdriver, which Chromium exposes as a known automation tell. Adds browse/src/stealth.ts: single source of truth for the stealth init script and launch args. Both browser-manager.launch() (headless) and launchHeaded() (persistent context with extension) call applyStealth(context) and pass STEALTH_LAUNCH_ARGS into chromium.launch. The pre-existing launchHeaded stealth that did fake plugins/languages is removed for the same reason. The cdc_/__webdriver runtime cleanup and Permissions API patch are kept — they remove automation-injected artifacts, not synthesize fake natural-browser values. Adds bridge-chromium-e2e.test.ts (codex F3): the test that proves the FEATURE works. Real Chromium with proxy.server = 'socks5://127.0.0.1: <bridgePort>' navigates to a local HTTP fixture; the auth upstream's connect counter and the HTTP fixture's hit counter both increment, proving traffic actually traversed bridge → auth-upstream → destination. Without this test, we could ship a working byte-relay and a broken Chromium integration and never know. Adds bridge-port-restart.test.ts (codex F1, reframed): old test assumed two daemons coexist, which contradicts D2 single-daemon model. Reframed as restart-then-restart, asserting fresh ephemeral ports (never the hardcoded 1090) on each spin-up. Adds stealth-webdriver.test.ts: navigator.webdriver=false in both fresh contexts and persistent contexts; navigator.plugins/languages are NOT replaced with the wintermute fake list (D7 verification). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(gstack): generate llms.txt — single-file capability index for AI agents Adds scripts/gen-llms-txt.ts: produces gstack/llms.txt at repo root, indexing every skill (47), every browse command (75), and design commands when the design CLI is present. Per the llmstxt.org convention, agents can read one file to learn what gstack offers instead of crawling 47 SKILL.md files. Sources: - skill SKILL.md.tmpl frontmatter (name + description block scalar) - browse/src/commands.ts COMMAND_DESCRIPTIONS (sorted by category) - design/src/commands.ts COMMAND_DESCRIPTIONS if present (best-effort) Wired into scripts/gen-skill-docs.ts as a post-step so it regenerates on every `bun run gen:skill-docs` (the same script that re-emits all SKILL.md files). Failures are non-fatal warnings, not build breaks — the generator never blocks SKILL.md regen. Strict mode (--strict, also used by tests) throws when a skill is missing name or description in its frontmatter, catching missing metadata before it ships. Tests: shape (top-level sections, sort order, single-line summary discipline), every-skill-and-command-appears, strict-mode rejection of incomplete frontmatter, and freshness check that the committed gstack/llms.txt matches what the generator produces now. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(browse): --navigate flag on download for browser-triggered files Adds the --navigate strategy from community PR garrytan#1355 (originally from @garrytan-agents). When set, download navigates to the URL with waitUntil:'commit' and captures the resulting browser download via page.waitForEvent('download'), then saves via download.saveAs(). Handles URLs that trigger files via Content-Disposition headers, multi-hop CDN redirects requiring browser cookies, or anti-bot CDN chains where page.request.fetch() can't follow the auth/redirect chain. Defaults still use the existing direct-fetch strategy. --navigate is opt-in. Goes through the same validateNavigationUrl SSRF gate as goto, so download --navigate cannot reach IPv4 metadata endpoints (AWS IMDSv1, GCP/Azure equivalents) or arbitrary internal hosts. Inferred content type from suggested filename for common extensions (epub, pdf, zip, gz, mp3/mp4, jpg/jpeg/png, txt, html, json) — falls back to application/octet-stream. Same 200MB cap as Strategy 1. Frames the use case generically (anti-bot CDN, Content-Disposition, redirect chains) rather than naming any specific site, per project voice rules. Co-Authored-By: @garrytan-agents Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: v1.28.0.0 — browse SKILL section + VERSION + CHANGELOG VERSION 1.27.1.0 → 1.28.0.0 (MINOR — substantial new capability: five new flags/features, ~600 LOC added, new socks dep, multiple new modules). browse/SKILL.md.tmpl: new "Headed Mode + Proxy + Anti-Bot Sites" section between User Handoff and Snapshot Flags. Documents --headed (auto-Xvfb on Linux), --proxy (with embedded SOCKS5 bridge for auth), download --navigate, the cred-mixing policy, daemon-discipline (refuse-on-mismatch), the narrowed webdriver-only stealth, container support caveats, and the fail-fast/no-retry failure modes. CHANGELOG entry follows the release-summary format from CLAUDE.md: two-line headline, lead paragraph, "The numbers that matter" table tied to specific test files that prove each capability, "What this means for AI agents" closing tied to a real workflow shift, then itemized Added/Changed/Fixed/For-contributors sections. Browse SKILL.md regenerated via bun run gen:skill-docs. gstack/llms.txt regenerated automatically from the same pipeline. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(browse): integration coverage for daemon mismatch + proxy fail-fast Adds two integration tests that exercise the full process boundary, not just the module-level wiring. daemon-mismatch-refuse.test.ts (D2): - Stubs a healthy state file with a fake configHash and a fake /health HTTP server, runs the actual cli.ts binary with a mismatching --proxy, asserts exit 1 + 'different config' / 'browse disconnect' hint in stderr. - Same shape with the plain-daemon-meets---headed case. - Positive case: matching configHash → CLI does NOT emit the mismatch hint (regardless of whether the actual command succeeds). server-proxy-fail-fast.test.ts: - Starts the rejecting SOCKS5 upstream, spawns server.ts with BROWSE_PROXY_URL pointing at it, BROWSE_HEADLESS_SKIP=1 to skip Chromium launch. - Asserts exit 1, 'FAIL upstream' in stderr (testUpstream pre-flight ran), no raw credential leakage in any output (redaction works on the failure path), and exit within 30s upper bound. Both tests use the existing spawn-bun-cli pattern from commands.test.ts so they run on the same CI infrastructure as the rest of the bun test suite. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(gen-skill-docs): keep module sync so test require() still works Two regressions caught by the full test suite after the v1.28.0.0 landing pass: 1) package.json version mismatch — VERSION was bumped to 1.28.0.0 but package.json still pinned to 1.27.1.0. test/gen-skill-docs.test.ts asserts they match. 2) Top-level await in scripts/gen-llms-txt.ts (CLI entry block) and scripts/gen-skill-docs.ts (post-step) made gen-skill-docs an async module. test/gen-skill-docs.test.ts uses require() to pull extractVoiceTriggers/processVoiceTriggers from gen-skill-docs, which Bun rejects on async modules with: "TypeError: require() async module ... unsupported. use 'await import()' instead." Fix: wrap the await blocks in void IIFEs so the modules remain sync from a require() perspective. After fix: all 379 gen-skill-docs tests pass, all 77 new feature tests pass (3 skipped on macOS — Linux+Xvfb gates). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(browse): apply codex adversarial findings on the new lifecycle Codex outside-voice review caught five real production-failure modes in the v1.28.0.0 proxy/headed lifecycle. Fixed: 1) `browse disconnect` skip-graceful for proxy-only daemons (browse/src/cli.ts). The graceful /command POST went out with stray `domains,` shorthand and (even fixed) the server's disconnect handler only tears down headed mode — proxy-only daemons returned 200 "Not in headed mode" while leaving the bridge running. Now disconnect short-circuits to force-cleanup for non-headed daemons, which kicks process.on('exit') in server.ts to close the bridge + Xvfb. 2) sendCommand crash retry preserves --proxy / --headed (browse/src/cli.ts). The ECONNRESET retry path called startServer() with no extraEnv, silently dropping the proxied flags. A daemon that died mid-command would silently restart in default direct/headless mode and bypass the SOCKS bridge. Now reapplies BROWSE_PROXY_URL, BROWSE_HEADED, and BROWSE_CONFIG_HASH from the resolved global flags. 3) `connect` honors --proxy (browse/src/cli.ts). The headed-mode `connect` command built its own serverEnv that didn't include BROWSE_PROXY_URL, so `browse --proxy <url> connect` launched headed Chromium without the proxy. Now threads proxyUrl + configHash into the connect serverEnv. 4) SOCKS5 bridge handles fragmented TCP frames (browse/src/socks-bridge.ts). Previously used once('data') and parsed each chunk as a complete SOCKS5 frame — TCP doesn't preserve message boundaries and split greetings/CONNECT requests caused intermittent handshake failures. Replaced with a single state machine that buffers chunks and uses size predicates on the SOCKS5 header to know when a complete frame has arrived. Pauses the client socket during upstream connect and replays any remainder bytes into the upstream on success. 5) Xvfb cleanup-then-state-delete ordering (browse/src/server.ts). emergencyCleanup() previously deleted the state file BEFORE any Xvfb cleanup could read it, orphaning Xvfb on uncaughtException / unhandledRejection. Now reads the state file first, calls cleanupXvfb() (which validates cmdline + start-time before kill), then deletes the state file. Adds a regression test for garrytan#4: writes the SOCKS5 greeting + CONNECT one byte at a time with 5ms ticks, asserts a clean round trip after the fragmented handshake. Codex's sixth finding (bridge advertises NO_AUTH on 127.0.0.1, so any co-located process can use the authenticated upstream) is documented as a known limitation — gstack's threat model assumes single-user hosts. Adding bridge-side auth is a separate change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: update BROWSER.md + TODOS.md for v1.28.0.0 BROWSER.md picks up a "Headed mode + proxy + browser-native downloads (v1.28.0.0)" subsection inside Real-browser mode plus the new source-map entries (socks-bridge.ts, proxy-config.ts, proxy-redact.ts, xvfb.ts, stealth.ts). TODOS.md anti-bot-stealth item updated to reflect the v1.28 narrowing — the "fake plugins" line is no longer accurate. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(ci): include bun.lock in image build for deterministic install CI evals all failed on PR garrytan#1363 with: error: Could not resolve: "smart-buffer". Maybe you need to "bun install"? error: Could not resolve: "ip-address". Maybe you need to "bun install"? at /opt/node_modules_cache/socks/build/client/socksclient.js:15 The cached node_modules layer in the pre-baked Docker image had `socks` (the new dep) but was missing its transitive deps (smart-buffer, ip-address). The image build copied only package.json into the build context — without bun.lock, `bun install` resolved a different tree than local `bun install` did, dropping required transitive deps. Reproduces locally as 229 packages (correct) when bun.lock is present or absent. Why CI diverged isn't fully understood — possibly Docker layer cache reuse across image rebuilds — but the deterministic fix is to include the lockfile in the image build context and use `--frozen-lockfile`, matching what every CI doc recommends. Changes: - .github/docker/Dockerfile.ci: COPY bun.lock alongside package.json, switch `bun install` → `bun install --frozen-lockfile` so any future lockfile drift fails loudly during image build instead of producing a partially-installed cache that breaks downstream eval jobs. - .github/workflows/evals.yml: include bun.lock in the image-tag hash so adding/removing a dep invalidates the image, AND copy bun.lock into the docker context alongside package.json. - .github/workflows/evals-periodic.yml: same updates. - .github/workflows/ci-image.yml: rebuild trigger now fires on bun.lock changes too; build context includes bun.lock. Image hash changes → fresh image gets built on next CI run → install matches the lockfile exactly → no missing transitive deps. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(ci): use hardlink copy instead of symlink for node_modules cache After the bun.lock fix landed, the eval matrix STILL failed identically: Could not resolve: "smart-buffer" / "ip-address" at /opt/node_modules_cache/socks/build/client/socksclient.js But the hash-tagged image actually contains smart-buffer + ip-address + socks all flat in /opt/node_modules_cache (verified by pulling and inspecting the image). 207 packages, all present. Root cause: the workflow used `ln -s /opt/node_modules_cache node_modules` to restore deps. Bun build (and Node module resolution generally) walks a file's realpath to find sibling deps. From the symlinked /workspace/node_modules/socks/build/client/socksclient.js, realpath resolves to /opt/node_modules_cache/socks/build/client/socksclient.js, and walking up to find a node_modules/smart-buffer dir fails — there's no `node_modules` segment in the realpath. Switch `ln -s` → `cp -al` (hardlink-copy). Each file in the cache becomes a hardlink at /workspace/node_modules/<pkg>, sharing inodes (no data copy). Realpath of /workspace/node_modules/socks/.../socksclient.js stays inside /workspace/node_modules, so sibling deps resolve correctly. Speed is comparable to symlink — `cp -al` on ~200 packages on tmpfs is sub-second. Same caching story preserved. Both evals.yml and evals-periodic.yml updated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(ci): cp -r instead of cp -al — /opt and /workspace are different filesystems The hardlink-copy fix landed and immediately broke with: cp: cannot create hard link 'node_modules/<file>' to '/opt/node_modules_cache/<file>': Invalid cross-device link GitHub Actions runners mount the workspace volume at /workspace (overlay-fs layered onto the runner image), and /opt is the runner image's own filesystem. Cross-filesystem hardlinks aren't supported. Switch `cp -al` → `cp -r`. Cost: ~5s for ~200 packages of small JS files vs ~0s for the broken symlink. Still cheaper than the ~15s `bun install` fallback. Realpath of /workspace/node_modules/<pkg>/... stays inside /workspace, so bun build's sibling-dep resolution works. Both evals.yml and evals-periodic.yml updated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Five features ship in one PR. Browse picks up
--proxy(with an embedded SOCKS5 bridge so Chromium can speak to authenticated upstreams it can't speak to natively),--headed(auto-spawns Xvfb on Linux containers without DISPLAY), anddownload --navigate(uses the browser's native download handler for Content-Disposition, multi-hop CDN redirects, and anti-bot CDN chains wherepage.request.fetch()falls over). Stealth is narrowed tonavigator.webdrivermasking only — modern fingerprinters punish inconsistent fakes, so faking plugins/languages was making detection easier, not harder.gstack/llms.txtis auto-generated from skill manifests + browse + design surfaces (47 skills + 75 commands in 11KB).Bisected commits: SOCKS5 bridge → CLI/server wiring → Xvfb → browser-manager stealth → llms.txt → PR #1355 merge → docs/CHANGELOG → integration tests → gen-skill-docs sync fix → codex follow-up fixes → BROWSER.md/TODOS.md doc sync.
Includes community PR #1355 from @garrytan-agents (
download --navigate) merged with attribution and site-specific names scrubbed for generic anti-bot framing.Test Coverage
Pre-Landing Review
Codex outside-voice flagged 6 production-failure modes. 5 fixed in commit 50d07eb (codex follow-up):
browse disconnectwas broken for proxy-only daemons (straydomains,in body + server only handled headed mode). Now skips graceful for non-headed and goes straight to force-cleanup.sendCommandcrash retry silently dropped--proxy/--headed— restart calledstartServer()with no extraEnv. Now reapplies the flags from resolved global state.connectignored--proxy— its own serverEnv didn't include the proxy. Fixed.once('data')assumed each chunk is a complete frame. Replaced with a state machine that buffers until SOCKS5 framing predicates say a complete frame is in. Regression test added.emergencyCleanupdeleted the state file before reading it for Xvfb cleanup. Now reads first, validates PID + start-time, kills, then deletes.The 6th finding (bridge advertises NO_AUTH on 127.0.0.1, so any co-located process can use the authenticated upstream) is documented as a known limitation — gstack's threat model assumes single-user hosts. Bridge-side auth is a separate change.
Eval Results
No prompt-related files changed — evals skipped.
Adversarial Review
Codex SAID (paraphrased): 6 findings (3 HIGH, 3 MEDIUM). Recommended "do not ship because the new proxy/headed lifecycle is currently broken in ways that can silently bypass the proxy and leave helper processes behind." All 6 walked through; 5 fixed, 1 documented.
Plan Completion
Verification Results
No dev server running locally — automated verification skipped. Manual smoke test (per CHANGELOG):
browse --headed --proxy $RES_SOCKS download "$URL" /tmp/file.bin --navigateagainst a residential SOCKS5 upstream + auth-required test URL. Lands the file with the expected magic bytes.TODOS
No previously-tracked TODOs completed by this branch. The pre-existing CDP-stealth TODO (rebrowser-patches) is updated to reflect that v1.28.0.0 narrowed stealth to
navigator.webdriveronly — that TODO remains open for the deeper CDP-protocol-level work.Documentation
Headed mode + proxy + browser-native downloads (v1.28.0.0)subsection inside Real-browser mode covering--headed,--proxy(SOCKS5 w/ auth via local bridge, HTTP/HTTPS pass-through),download --navigate, credential policy, daemon discipline, narrowed stealth scope (navigator.webdriveronly), Linux container/Xvfb support, and failure modes. Source map picks upsocks-bridge.ts,proxy-config.ts,proxy-redact.ts,xvfb.ts,stealth.ts. TOC anchor added.Anti-bot stealth: Playwright CDP patchesitem to reflect the v1.28.0.0 narrowing of stealth.Test plan
bun run scripts/gen-skill-docs.tsregenerates clean (47 skills, 75 browse commands in llms.txt)bun run scripts/gen-llms-txt.ts --dry-runconfirms freshnessbrowse --headed --proxy $RES_SOCKS download <auth-required URL> /tmp/file.bin --navigate— pending real-world rerun by reviewer🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmithwith what you need.