ci: make TruffleHog output actionable and drop the Lob false positive - #263
Merged
Ruari-Phipps merged 1 commit intoAug 10, 2026
Conversation
The secret-scanning gate failed on PR polyai#259 with a single line: Found verified Lob result 🐷🔑 No file, no line, no match. The trigger turned out to be a Python test function name, src/poly/tests/metrics_test.py:321: def test_add_duplicate_metric_friendly_error(...) Two independent problems. 1. The finding was noise. TruffleHog's Lob detector uses the pattern `\b((live|test)_[a-zA-Z0-9_]{35})\b` with no keyword-proximity requirement, and its Keywords() prefilter is literally {"live_", "test_"} — the pytest naming convention. So every chunk in the test suite reaches an ungated regex that matches any 40-character `test_*` identifier. It also counts HTTP 403 and 422 from api.lob.com as "verified", which is how a function name came to be reported as a verified secret. Scanning all of main's history with 3.96.0 returns 85 findings, every one of them Lob; excluding the detector takes that to zero. main already contains 36 more identifiers that match, so this would have recurred on any PR touching those lines. ADK does not use Lob, so --exclude-detectors=lob costs no coverage. The other 19 ungated detectors whose prefilter keywords appear in this repo (twilio, uri, redis, ftp, mongodb, launchdarkly, sendgrid, salesforce and friends) were checked against the source tree and have no matches and no near-misses. 2. The output was unactionable regardless. The action always passes --github-actions, whose printer emits only the detector name and verification status; file and line go into annotation metadata that never renders in the log, and the match is never printed at all. Since --json outranks --github-actions in TruffleHog's printer selection, a diagnostic step now re-runs the same scan on failure and reports file, line, commit, author and a 12-character match prefix — enough to triage, not enough to use. That step exits 0; the gate above has already failed the job. Also pins the scanner image. The action was SHA-pinned but its `version` input defaulted to `latest`, so the binary doing the scanning floated independently of the pin.
northdpole
force-pushed
the
spyros/trufflehog-verbose-output-and-lob-fp
branch
from
August 7, 2026 16:55
d4ebb43 to
7277b36
Compare
7 tasks
Ruari-Phipps
approved these changes
Aug 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What happened
The secret-scanning gate failed on #259 with exactly one line of output:
No file, no line, no match. The actual trigger was a Python test function name —
src/poly/tests/metrics_test.py:321:That's two separate problems, and this PR fixes both.
1. The finding was noise
TruffleHog's Lob detector combines two individually-weak choices:
The
Keywords()prefilter is the pytest naming convention, so every chunk in our test suite reaches an ungated regex that matches any 40-charactertest_*identifier. It also treats HTTP 403 and 422 fromapi.lob.comas "verified" — only 401 counts as invalid — which is how a function name got reported as a verified secret.Measured against this repo with TruffleHog 3.96.0 (
--no-verification, all result types):--exclude-detectors=lobmainhistoryLob is 100% of the finding surface.
mainalready contains 36 more identifiers that match the pattern — they only stay quiet because scans are limited to each PR's commit range, so this would have recurred on any PR touching those lines. Roughly 4% of test names in the repo land on exactly 40 characters.ADK does not use Lob (direct-mail API), so there is no coverage to lose.
Other detectors were checked, not assumed
Of 870 detectors, 133 lack a keyword-proximity gate and 74 have a prefilter keyword present in this repo. Intersecting gives 19 others that are live and ungated. All were tested against the source tree — zero matches and zero near-misses:
Two worth noting: Twilio (
AC+32hex) is the one to watch given our telephony code — clean today, but a realistic-shaped dummy SID in a fixture will fire it. Auth0 looks alarming (\b(ey[a-zA-Z0-9._-]+)\bwith prefilter keywordstoken/domain) butFromDatarequires a 2000–5000 char match paired with an*.auth0.comdomain, so it's well gated.2. The output was unactionable regardless
The action always passes
--github-actions, and that printer emits only the detector name and verification status:File and line go into annotation metadata that never renders in the log, and the match is never printed at all.
extra_argscan't remove--github-actions, but--jsonoutranks it in TruffleHog's printer selection, so a diagnostic step re-runs the scan on failure:Only a 12-character prefix is printed — enough to triage, not enough to use. The step exits 0; the gate above has already failed the job.
3. Also: the scanner version was floating
The action was SHA-pinned, but its
versioninput defaulted tolatest— the failing run logsVERSION: latest. The pin covered the wrapper, not the binary doing the scanning. Now pinned tov3.96.0.Testing
jqfilter tested against a real TruffleHog JSON result record.mainhistory to zero findings.--exclude-detectorsso the two can't disagree.Note for reviewers
This repo is public, and the description above documents which detector is disabled and that
--results=verified,unknownfilters out unverifiable detectors (jwt,uri,sqlserver). It's all derived from TruffleHog's public source. Happy to trim if anyone would rather that analysis live internally.