feat: implicit viewport grab + CLI peek command#464
feat: implicit viewport grab + CLI peek command#464devin-ai-integration[bot] wants to merge 3 commits into
Conversation
When React Grab is idle (not actively selecting), automatically collect visible elements in the viewport and write them to the clipboard on pointermove/scroll (debounced 3s). The structured payload uses both text/plain and the web custom MIME type so the existing watcher daemon recognizes it. Viewport grabs carry an implicit:true flag to distinguish them from explicit user selections. Also adds a 'peek' CLI command that reads the clipboard once (no daemon needed) and prints any grab payload it finds, enabling one-shot reads for skills that only need the current viewport context. Co-Authored-By: Aiden Bai <aiden.bai05@gmail.com>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
commit: |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| } | ||
| }, | ||
| { passive: true }, | ||
| ); |
There was a problem hiding this comment.
Scroll listener misses nested scrollers
Medium Severity
The new scroll handler is registered on window with { passive: true } only. scroll events do not bubble, so this listener misses scrolling inside nested overflow containers. Viewport clipboard updates then rely on pointermove, leaving implicit context stale after in-panel scroll.
Reviewed by Cursor Bugbot for commit ac22acf. Configure here.
|
|
||
| const resolveElementEntry = async (element: Element): Promise<ViewportEntry | null> => { | ||
| const fiber = getFiberFromHostInstance(element); | ||
| if (!fiber) return null; |
There was a problem hiding this comment.
Fiber lookup skips parent walk
Medium Severity
resolveElementEntry calls getFiberFromHostInstance on the sampled DOM node only. Elsewhere, grabs resolve via findNearestFiberElement, walking ancestors until a host fiber exists. Sampled leaf nodes without their own fiber are dropped, so implicit viewport lists can omit elements explicit copy would include.
Reviewed by Cursor Bugbot for commit ac22acf. Configure here.
| let isWriting = false; | ||
|
|
||
| const performWrite = async (): Promise<void> => { | ||
| if (isWriting) return; |
There was a problem hiding this comment.
Concurrent write skips scheduled flush
Low Severity
When performWrite is already running, another debounced invocation returns immediately at if (isWriting) return with no follow-up. Long getSource batches can cause a timer-fired update to be dropped until the user moves or scrolls again.
Reviewed by Cursor Bugbot for commit ac22acf. Configure here.
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
|
|
||
| const formatEntryReference = (entry: ViewportEntry): string => { | ||
| const tag = entry.componentName | ||
| ? `<${entry.tagName}>` |
There was a problem hiding this comment.
formatEntryReference displays the underlying HTML element tag instead of the component name when a component wraps the element
Co-Authored-By: Aiden Bai <aiden.bai05@gmail.com>
| () => { | ||
| if (!isActivated() && !isHoldingKeys() && !isCopying()) { | ||
| viewportClipboard.scheduleWrite(); | ||
| } |
There was a problem hiding this comment.
Idle grab overwrites explicit copy
Medium Severity
Implicit scheduleWrite runs whenever the session is not active, not holding, and not copying. After an explicit grab finishes, state becomes justCopied then often inactive, so pointer or scroll activity can debounce a viewport write that replaces the user’s structured clipboard payload.
Reviewed by Cursor Bugbot for commit 47b95b6. Configure here.
| } | ||
| } catch {} finally { | ||
| isWriting = false; | ||
| } |
There was a problem hiding this comment.
Write may run after dispose
Low Severity
dispose only calls viewportClipboard.cancel(), which clears the debounce timer. An in-flight performWrite can still finish afterward and call navigator.clipboard.write, updating the clipboard after React Grab has torn down.
Reviewed by Cursor Bugbot for commit 47b95b6. Configure here.
| return undefined; | ||
| }; | ||
|
|
||
| const resolveElementEntry = async (element: Element): Promise<ViewportEntry | null> => { |
There was a problem hiding this comment.
Implicit viewport grabs skip elements when the sampled point lands on a leaf DOM node without a React fiber, whereas explicit grabs would include those elements by walking up the DOM tree
| const location = locationParts.length > 0 ? ` ${locationParts.join(" ")}` : ""; | ||
| return `[${tag}${location}]`; | ||
| }; | ||
|
|
There was a problem hiding this comment.
Clipboard write operations are called from a setTimeout callback outside the user gesture context, causing modern browsers to reject them with NotAllowedError
| let timerId: ReturnType<typeof setTimeout> | null = null; | ||
| let lastContentHash = ""; | ||
| let isWriting = false; | ||
|
|
There was a problem hiding this comment.
Viewport changes are dropped when a new debounced write is scheduled while a previous write is still in progress
| viewportClipboard.scheduleWrite(); | ||
| } | ||
| }, | ||
| { passive: true }, |
There was a problem hiding this comment.
Scroll event listener on window needs capture: true to catch scroll events from nested overflow: auto containers
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| if (!isActivated() && !isHoldingKeys() && !isCopying()) { | ||
| viewportClipboard.scheduleWrite(); | ||
| } |
There was a problem hiding this comment.
🔴 Viewport clipboard writes continue when react-grab is explicitly disabled
The viewportClipboard.scheduleWrite() calls in both the pointermove and scroll handlers only gate on !isActivated() && !isHoldingKeys() && !isCopying(), but never check isEnabled(). When a user explicitly disables react-grab (via the toolbar or setEnabled(false) API), all three conditions are trivially true (activation is blocked when disabled), so the viewport clipboard keeps writing to the user's clipboard. This is inconsistent with the rest of the codebase: the keydown handler checks isEnabled() at index.tsx:2352, the toolbar action handler at index.tsx:1528, handleComment at index.tsx:1577, and redetectElementUnderPointer at index.tsx:2798.
| if (!isActivated() && !isHoldingKeys() && !isCopying()) { | |
| viewportClipboard.scheduleWrite(); | |
| } | |
| if (!isActivated() && !isHoldingKeys() && !isCopying() && isEnabled()) { | |
| viewportClipboard.scheduleWrite(); | |
| } |
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Fixed in b3b620e — added isEnabled() guard to both pointermove and scroll handlers.
| if (!isActivated() && !isHoldingKeys() && !isCopying()) { | ||
| viewportClipboard.scheduleWrite(); | ||
| } |
There was a problem hiding this comment.
🔴 Same missing isEnabled() guard on the scroll handler's viewport clipboard write
The scroll event handler at index.tsx:2605 has the same missing isEnabled() check as the pointermove handler, allowing viewport clipboard writes to continue when react-grab is disabled.
| if (!isActivated() && !isHoldingKeys() && !isCopying()) { | |
| viewportClipboard.scheduleWrite(); | |
| } | |
| if (!isActivated() && !isHoldingKeys() && !isCopying() && isEnabled()) { | |
| viewportClipboard.scheduleWrite(); | |
| } |
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Fixed in b3b620e — same isEnabled() guard added here.
| const record = { | ||
| source: "text", | ||
| timestamp: Date.now(), | ||
| content: snapshot.text, | ||
| entries: [], | ||
| implicit: false, | ||
| id: `peek-${Date.now()}`, | ||
| receivedAt: Date.now(), | ||
| }; |
There was a problem hiding this comment.
🟡 Inconsistent timestamps in text-fallback record due to separate Date.now() calls
In the text-fallback path, timestamp (line 83), the timestamp embedded in id (line 87), and receivedAt (line 88) each call Date.now() independently, so they can produce different values. This is inconsistent with the structured-grab path (lines 66-72) where parsedGrab.timestamp is reused for both timestamp and id. A downstream consumer matching id against timestamp would see a mismatch.
Text-fallback path with three separate Date.now() calls
const record = {
source: "text",
timestamp: Date.now(), // Call 1
content: snapshot.text,
entries: [],
implicit: false,
id: `peek-${Date.now()}`, // Call 2 — may differ
receivedAt: Date.now(), // Call 3 — may differ
};| const record = { | |
| source: "text", | |
| timestamp: Date.now(), | |
| content: snapshot.text, | |
| entries: [], | |
| implicit: false, | |
| id: `peek-${Date.now()}`, | |
| receivedAt: Date.now(), | |
| }; | |
| const record = { | |
| source: "text", | |
| timestamp: Date.now(), | |
| content: snapshot.text, | |
| entries: [], | |
| implicit: false, | |
| id: `peek-${Date.now()}`, | |
| receivedAt: Date.now(), | |
| }; |
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Fixed in b3b620e — now captures Date.now() once into now and reuses it for timestamp, id, and receivedAt.
| const writeToClipboard = async (content: string, metadata: string): Promise<boolean> => { | ||
| if (!navigator.clipboard) return false; | ||
|
|
||
| try { | ||
| if (typeof ClipboardItem !== "undefined") { | ||
| const textBlob = new Blob([content], { type: "text/plain" }); | ||
| const customBlob = new Blob([metadata], { type: REACT_GRAB_MIME_TYPE }); | ||
| const item = new ClipboardItem({ | ||
| "text/plain": textBlob, | ||
| [WEB_CUSTOM_MIME_TYPE]: customBlob, | ||
| }); | ||
| await navigator.clipboard.write([item]); | ||
| return true; | ||
| } | ||
| } catch {} | ||
|
|
||
| try { | ||
| await navigator.clipboard.writeText(content); | ||
| return true; | ||
| } catch {} |
There was a problem hiding this comment.
🚩 Clipboard API writes will silently fail in most contexts without user activation
The writeToClipboard function in viewport-clipboard.ts:91-110 uses navigator.clipboard.write() and navigator.clipboard.writeText(), both of which require transient user activation in most modern browsers. The scheduleWrite is triggered from pointermove and scroll events (which do not grant user activation) and debounced by 3 seconds (VIEWPORT_CLIPBOARD_DEBOUNCE_MS), meaning any activation from a qualifying event (like pointerdown) would have expired. The calls are wrapped in empty catch blocks so failures are silent. In Chrome on localhost (common for React development), the clipboard-write permission may be auto-granted, making this work. On Firefox/Safari or in production HTTPS contexts, this feature will be a silent no-op. Whether this is acceptable depends on the target deployment scenario.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
This is by design — the feature targets Chrome on localhost (where clipboard-write is auto-granted), which covers the primary dev-tool use case. The empty catches ensure silent degradation on Firefox/Safari/production HTTPS contexts. No action needed.
…timestamps in peek Co-Authored-By: Aiden Bai <aiden.bai05@gmail.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 6 total unresolved issues (including 5 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit b3b620e. Configure here.
| timerId = setTimeout(() => { | ||
| timerId = null; | ||
| void performWrite(); | ||
| }, VIEWPORT_CLIPBOARD_DEBOUNCE_MS); |
There was a problem hiding this comment.
Debounced write overwrites grabs
High Severity
Implicit viewport writes are scheduled while React Grab is idle, but performWrite runs after the debounce without checking whether the user is still idle. A pending timer is not cancelled when activation, key-hold, or copying starts, so an explicit grab on the clipboard can be replaced seconds later by a stale viewport snapshot.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit b3b620e. Configure here.
Testing ResultsTested the viewport grab feature end-to-end on the e2e-app (localhost:5175) with browser GUI + CLI shell tests. Feature Tests (6/6 passed)
CI Status
Notes
|
1 participant
Summary
Adds "magical implicit grab" — when React Grab is idle, it silently writes visible viewport elements to the clipboard on
pointermove/scroll(debounced 3 s), so agents running the react-grab skill get page context without the user explicitly selecting anything.Browser side (
packages/react-grab/):collectViewportElements()— samples a 6×6 grid across the viewport viadocument.elementsFromPoint, filters throughisValidGrabbableElement, deduplicates, caps at 30 elements.createViewportClipboard()— debounced controller that resolves each element's fiber → component name + source location, formats[<tag> in Component (at file:line:col)]references, writes to clipboard vianavigator.clipboard.write()with bothtext/plainandweb application/x-react-grab(falls back towriteText). Skips redundant writes by comparing content hash. Payload carriesimplicit: trueto distinguish from explicit grabs.pointermove+ newscrolllistener incore/index.tsx— only fires when not in active/holding/copying state. Cleanup ondispose().CLI side (
packages/cli/):peekcommand — one-shot clipboard read (no daemon). Reads the structured grab or text fallback, prints one JSON line if a grab is found, exits. Lets skills donpx react-grab peekfor a non-blocking viewport snapshot instead of entering thepullwatch loop.Link to Devin session: https://app.devin.ai/sessions/e65f397944e2488f81b02b25e63c2da1
Requested by: @aidenybai
Note
Medium Risk
Background clipboard writes on scroll/pointermove can surprise users or overwrite clipboard content, but only when the tool is idle and writes are debounced/deduplicated; peek is read-only CLI surface area.
Overview
Adds implicit viewport context while React Grab is idle: on debounced
pointermoveandscroll, the extension samples visible DOM nodes, resolves React source metadata, and writes a grab payload to the clipboard withimplicit: true(without activating the overlay).Browser (
react-grab): NewcollectViewportElements(6×6 grid, capped at 30 grabbable elements) andcreateViewportClipboard(3s debounce, dedupe, skip unchanged content,text/plain+ custom MIME). Hooked from core when enabled and not active/holding/copying; cancelled on dispose.CLI: New
peekcommand — one-shot clipboard read (no watch daemon), prints a single JSON line for structured grabs or grab-like text (source: viewport/custom/text), otherwise exits silently.Reviewed by Cursor Bugbot for commit b3b620e. Bugbot is set up for automated code reviews on this repo. Configure here.
Summary by cubic
Adds implicit viewport grabbing in
packages/react-grabto auto-snapshot visible elements when idle, and a newpeekcommand inpackages/clifor one-shot clipboard reads without the daemon. This gives agents lightweight page context with no user action.New Features
text/plainandweb application/x-react-graband setsimplicit: true.peek:npx react-grab peekreads the clipboard once (native or--text-only), prints a single JSON record if a grab is found, then exits. Includessource(viewport/custom/text),implicit,timestamp,entries,content,id, andreceivedAt.Bug Fixes
isEnabled(); ensure consistent timestamps inpeekoutput.Written for commit b3b620e. Summary will update on new commits.