Skip to content

ci: make TruffleHog output actionable and drop the Lob false positive - #263

Merged
Ruari-Phipps merged 1 commit into
polyai:mainfrom
northdpole:spyros/trufflehog-verbose-output-and-lob-fp
Aug 10, 2026
Merged

ci: make TruffleHog output actionable and drop the Lob false positive#263
Ruari-Phipps merged 1 commit into
polyai:mainfrom
northdpole:spyros/trufflehog-verbose-output-and-lob-fp

Conversation

@northdpole

Copy link
Copy Markdown
Contributor

What happened

The secret-scanning gate failed on #259 with exactly one line of output:

##[warning]Found verified Lob result 🐷🔑

No file, no line, no match. The actual trigger was a Python test function name — src/poly/tests/metrics_test.py:321:

def test_add_duplicate_metric_friendly_error(self, mock_load, mock_create, mock_error):

That's two separate problems, and this PR fixes both.

1. The finding was noise

TruffleHog's Lob detector combines two individually-weak choices:

keyPat = regexp.MustCompile(`\b((live|test)_[a-zA-Z0-9_]{35})\b`)   // no keyword-proximity gate
func (s Scanner) Keywords() []string { return []string{"live_", "test_"} }

The Keywords() prefilter is the pytest naming convention, so every chunk in our test suite reaches an ungated regex that matches any 40-character test_* identifier. It also treats HTTP 403 and 422 from api.lob.com as "verified" — only 401 counts as invalid — which is how a function name got reported as a verified secret.

Measured against this repo with TruffleHog 3.96.0 (--no-verification, all result types):

Scope Findings After --exclude-detectors=lob
Working tree 51 (all Lob) 0
Full main history 85 (all Lob) 0

Lob is 100% of the finding surface. main already contains 36 more identifiers that match the pattern — they only stay quiet because scans are limited to each PR's commit range, so this would have recurred on any PR touching those lines. Roughly 4% of test names in the repo land on exactly 40 characters.

ADK does not use Lob (direct-mail API), so there is no coverage to lose.

Other detectors were checked, not assumed

Of 870 detectors, 133 lack a keyword-proximity gate and 74 have a prefilter keyword present in this repo. Intersecting gives 19 others that are live and ungated. All were tested against the source tree — zero matches and zero near-misses:

uri (user:pass@host)  0    twilio AC<32hex>   0    stripe pi_..._secret_  0
redis:// creds        0    twilio SK<32>      0    zohocrm 1000.hex.hex   0
ftp:// creds          0    launchdarkly       0    sendgrid SG.           0
mongodb:// creds      0    salesforce         0    mailchimp <32hex>-usN  0
gemini master-/account- 0  closecrm api_<45>  0    lob (contrast)        43

Two worth noting: Twilio (AC+32hex) is the one to watch given our telephony code — clean today, but a realistic-shaped dummy SID in a fixture will fire it. Auth0 looks alarming (\b(ey[a-zA-Z0-9._-]+)\b with prefilter keywords token/domain) but FromData requires a 2000–5000 char match paired with an *.auth0.com domain, so it's well gated.

2. The output was unactionable regardless

The action always passes --github-actions, and that printer emits only the detector name and verification status:

fmt.Printf("::warning file=%s,line=%d,endLine=%d::%s", out.Filename, out.StartLine, out.StartLine, message)
// message = "Found verified Lob result 🐷🔑"

File and line go into annotation metadata that never renders in the log, and the match is never printed at all. extra_args can't remove --github-actions, but --json outranks it in TruffleHog's printer selection, so a diagnostic step re-runs the scan on failure:

──────────────────────────────────────────
detector: Lob [verified]
file:     src/poly/tests/metrics_test.py
line:     321
commit:   0b99540672d0 by Bill <bill@poly-ai.com>
match:    test_add_dup… (40 chars)

Only a 12-character prefix is printed — enough to triage, not enough to use. The step exits 0; the gate above has already failed the job.

3. Also: the scanner version was floating

The action was SHA-pinned, but its version input defaulted to latest — the failing run logs VERSION: latest. The pin covered the wrapper, not the binary doing the scanning. Now pinned to v3.96.0.

Testing

  • YAML validated.
  • jq filter tested against a real TruffleHog JSON result record.
  • Exclusion confirmed to take both the working tree and full main history to zero findings.
  • Diagnostic step mirrors the gate's --exclude-detectors so the two can't disagree.

Note for reviewers

This repo is public, and the description above documents which detector is disabled and that --results=verified,unknown filters out unverifiable detectors (jwt, uri, sqlserver). It's all derived from TruffleHog's public source. Happy to trim if anyone would rather that analysis live internally.

@northdpole
northdpole requested a review from a team August 7, 2026 16:53
The secret-scanning gate failed on PR polyai#259 with a single line:

    Found verified Lob result 🐷🔑

No file, no line, no match. The trigger turned out to be a Python test
function name, src/poly/tests/metrics_test.py:321:

    def test_add_duplicate_metric_friendly_error(...)

Two independent problems.

1. The finding was noise. TruffleHog's Lob detector uses the pattern
   `\b((live|test)_[a-zA-Z0-9_]{35})\b` with no keyword-proximity
   requirement, and its Keywords() prefilter is literally {"live_",
   "test_"} — the pytest naming convention. So every chunk in the test
   suite reaches an ungated regex that matches any 40-character `test_*`
   identifier. It also counts HTTP 403 and 422 from api.lob.com as
   "verified", which is how a function name came to be reported as a
   verified secret.

   Scanning all of main's history with 3.96.0 returns 85 findings, every
   one of them Lob; excluding the detector takes that to zero. main
   already contains 36 more identifiers that match, so this would have
   recurred on any PR touching those lines. ADK does not use Lob, so
   --exclude-detectors=lob costs no coverage.

   The other 19 ungated detectors whose prefilter keywords appear in this
   repo (twilio, uri, redis, ftp, mongodb, launchdarkly, sendgrid,
   salesforce and friends) were checked against the source tree and have
   no matches and no near-misses.

2. The output was unactionable regardless. The action always passes
   --github-actions, whose printer emits only the detector name and
   verification status; file and line go into annotation metadata that
   never renders in the log, and the match is never printed at all.
   Since --json outranks --github-actions in TruffleHog's printer
   selection, a diagnostic step now re-runs the same scan on failure and
   reports file, line, commit, author and a 12-character match prefix —
   enough to triage, not enough to use. That step exits 0; the gate above
   has already failed the job.

Also pins the scanner image. The action was SHA-pinned but its `version`
input defaulted to `latest`, so the binary doing the scanning floated
independently of the pin.
@northdpole
northdpole force-pushed the spyros/trufflehog-verbose-output-and-lob-fp branch from d4ebb43 to 7277b36 Compare August 7, 2026 16:55
@Ruari-Phipps
Ruari-Phipps merged commit 8ff6342 into polyai:main Aug 10, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants