Add attester package-verification instruction and import-check hook 🤖🤖🤖#2387
Open
maminihds wants to merge 1 commit into
Open
Add attester package-verification instruction and import-check hook 🤖🤖🤖#2387maminihds wants to merge 1 commit into
maminihds wants to merge 1 commit into
Conversation
The instruction tells Copilot to verify PyPI/npm package and symbol names against the attester.dev existence oracle (free keyless tier, 25 calls/day per client IP) before installing or importing, catching hallucinated dependency names (5.2-21.7% of LLM suggestions per USENIX Security 2025). The preToolUse hook enforces the same check on code the agent writes: stdlib-only script, blocks only on a confident oracle negative, fails open on quota or network trouble, caches answers on disk. README tables regenerated with npm start.
github-actions
Bot
added
hooks
Contributor
|
🟡 Contributor Reputation Check: MEDIUM risk
Maintainers: please review this contributor before merging. |
github-actions
Bot
added
the
needs-review:MEDIUM
Contributor
🔒 PR Risk Scan ResultsScanned 6 changed file(s).
✅ No matching risk patterns were detected in changed files.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
Two entries for the same gap, a failure mode models cannot self-check:
hallucinated package names.
instructions/attester-verify-packages.instructions.md: tellsCopilot to verify PyPI/npm package and symbol names against the
attester.dev existence oracle before installing or importing.
hooks/attester-import-check/: apreToolUsehook that enforcesthe same check on code the coding agent writes. Stdlib-only Python
(3.10+, no dependencies), executable, documented in its README.
Why
A USENIX Security 2025 study measured 5.2% to 21.7% of LLM-suggested
package names as nonexistent. Attackers register those names
("slopsquatting"). Retrieval tools cannot answer "does this name exist at
all"; the oracle answers from real published artifacts (PyPI wheels, npm
tarballs) and signs every negative.
Cost to users
None required. The instruction and hook use the free keyless tier: 25
calls/day per client IP, no account or API key. The hook caches answers on
disk (exists 30 days, negatives 1 day) so routine editing rarely spends
quota, and it fails open with a warning when the quota or the network is
gone: it can never block a session on infrastructure trouble.
Disclosure
I built and run attester.dev, the service behind the check. The free tier
is the default path in both artifacts; a paid route for higher volume
exists and is mentioned in one line at the end of the instruction. Happy
to trim that mention further if it conflicts with the paid-services
guidance in discussions/968.
Tested
npm startregenerates the README tables cleanly (hook row +instruction row present).
(exit 1), real names allowed (exit 0), malformed input allowed (exit 0).