Skip to content

fix(guardrail): do not block prose whose words fuse into a blocklist entry - #177

Closed
rwagwani wants to merge 4 commits into
NVIDIA:mainfrom
rwagwani:fix/guardrail-blocklist-word-join-false-positive
Closed

fix(guardrail): do not block prose whose words fuse into a blocklist entry#177
rwagwani wants to merge 4 commits into
NVIDIA:mainfrom
rwagwani:fix/guardrail-blocklist-word-join-false-positive

Conversation

@rwagwani

@rwagwani rwagwani commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator

fix(guardrail): do not block prose whose words fuse into a blocklist entry

Stacked on #176. This branch contains that commit as its parent, so the diff shown here
is only the second commit (a46df92). Merge #176 first, or merge this one and #176 becomes
redundant. The two defects are independent; they were split so each can be reviewed on its
own.

What is wrong

better_profanity concatenates adjacent words with their separators removed, so that a
blocked word written with a space inserted mid-word is still caught — "n ike" for the entry
nike:

# better_profanity/utils.py -- any_next_words_form_swear_word
full_word = "%s%s" % (full_word, single_word.lower())          # separators dropped
if full_word in censor_words or full_word_with_separators in censor_words:
    return True, end_index

The side effect is that ordinary prose collides with short blocklist entries. "a desk in the background" fuses into deskin, which is on the blocklist alongside deskinned and
deskinning, so a sentence describing office furniture is blocked as gore.

This is not governed by guardrail_partial_match_min_chars; that parameter applies to
check_partial_match against the exact-match list, which is a different code path.

Observed in an Edge reasoner QA sweep: item 4_3, whose output describes "a robot standing
in a library, carrying a stack of books, with a desk in the background".

The fix

Before blocking, re-check the flagged prompt with word boundaries respected. A match counts
as legitimate when:

  • a single whitespace-delimited token matches on its own — this keeps joins inside a
    token, such as "desk-in", blocking;
  • a run of tokens matches a blocklist entry that genuinely contains spaces, such as
    "Boston Dynamics";
  • a run of tokens matches only once the spaces are deleted and at least one part is not
    ordinary English — the shape of a deliberate evasion.

A space-deleted join whose every part is an ordinary word is rejected as coincidence.

Single letters do not count as ordinary English even though WordNet knows them, because a
lone letter beside another word is what an evasion looks like, not normal writing. That keeps
"n ike" blocking.

Matching is delegated to better_profanity throughout, so leet substitutions and punctuation
behave exactly as before. WordNet is consulted only to classify a word as ordinary English;
if the corpus is unavailable the answer is "not ordinary", which keeps the filter at least as
strict as it is today.

Behaviour

input before after
a desk in the background BLOCKED SAFE
a desk in front of a bookshelf BLOCKED SAFE
n ike shoes (evasion) BLOCKED BLOCKED
to yota cars (evasion) BLOCKED BLOCKED
desk-in the corner (punctuation join) BLOCKED BLOCKED
a Boston Dynamics robot (multi-word entry) BLOCKED BLOCKED
the Nike logo (single-word entry) BLOCKED BLOCKED
deskin (the entry itself) BLOCKED BLOCKED

Swept over all 492 outputs of the Edge reasoner QA corpus: blocks drop from 11 to the 3
genuine trademark hits
(INTEL, Nike, Huracan), with no new blocks. The other five
of the original eleven are cleared by #176.

Cost

The re-check runs only after something has already matched, so text that was never flagged
costs nothing. On a flagged 106-token output the added work is ~43 ms, against ~340 ms
for the surrounding is_safe() call, which is dominated by the pre-existing nltk tokenise
and lemmatise pass.

Tests

Ten regression tests added, covering the fused-prose false positive, split evasions,
single-letter splits, multi-word phrases, punctuation joins, leet spellings of multi-word
entries, the join window's bound, leet characters inside tokens, and the strict-mode opt-out.
They inject the dictionary instead of reading WordNet, so they are hermetic — no checkpoint
download, no nltk corpus — and they exercise the heuristic directly rather than through
whatever the corpus happens to contain.

The helper builds its state by calling _configure_join_bookkeeping(), the same method
__init__ uses, so the join bookkeeping under test is production's derivation rather than a
copy of it.

$ pytest cosmos_framework/auxiliary/guardrail/blocklist/blocklist_test.py -q
18 passed

One of these tests caught a real bug during development: the join window was initially bounded
by the longest blocklist entry's word count, which is 1 for a single-word list, so pairs were
never examined. It passed against the production list only because that list happens to
contain a 6-word phrase. Join detection now carries its own bound.

Escape class, stated plainly

This narrows evasion detection, and the trade is documented in _has_legitimate_match rather
than left implicit:

  • Introduced here. A banned word split into pieces that are all multi-letter dictionary
    words is no longer treated as an evasion — "assassin" written as "ass ass in" would pass.

  • CORRECTED. An earlier revision of this section claimed fused matching "never reached far
    to begin with" — that with the entry nike, stock blocked "n ike" and "ni ke" but not
    "nik e" or "n i k e". That was measured on fragments with no trailing token, and it is
    wrong. The library's window needs a following word to close; once there is one, stock blocks
    all four forms:

    input stock with this PR
    n ike shoe blocked blocked
    ni ke shoe blocked blocked
    nik e shoe blocked blocked
    n i k e shoe blocked blocked

    So stock is stricter than this PR originally described, and the exemption below gives up more
    than was first stated. Measured against the production WordNet corpus, all four rows still
    block with this PR — but only because WordNet has no synset for ke, nik, k alone as a
    fused part, and so on. That is the trust boundary in the next bullet doing the work, and it is
    the honest reason those rows survive, not a property of the heuristic itself.

  • WordNet is the trust boundary. "Ordinary English" means "WordNet has a synset", which
    over-includes: it counts single letters such as n and f as words. That is why
    _is_prose_word requires two characters — without it, "n ike" would escape.

  • Layering. The blocklist is a coarse pre-filter running alongside model-based guardrails
    (llamaGuard3, qwen3guard, video content safety). It is not the last line of defence,
    which is what makes this trade reasonable rather than reckless.

Opt-out

guardrail_exempt_fused_prose switches the behaviour. Set it False to restore the stricter
previous behaviour: ordinary prose is blocked again, but no fused match is ever let through.
Verified both ways over the QA corpus — the default blocks the 3 genuine trademark hits, strict
mode adds 4_3 back.

Because presets.py:17 constructs Blocklist() with no arguments, the flag also reads
COSMOS_GUARDRAIL_EXEMPT_FUSED_PROSE when it is not passed explicitly, so strict matching is
selectable at deploy time rather than only by editing source:

COSMOS_GUARDRAIL_EXEMPT_FUSED_PROSE=0    # strict: block every fused match

That closes the "no off switch in practice" half of the strictness question. The other half —
whether the relaxed setting should be the default for every caller at all — is a decision for
the guardrail owners, not something this PR should settle on its own. Flagging it explicitly
rather than letting it ride in on a false-positive fix.

If the maintainers would rather not carry a heuristic here at all, there is a cleaner option
worth considering instead of this PR: fix the data. deskin is a rare gore stem that is
also the fusion of two of the commonest English words, sitting on a list that is otherwise
trademarks. Pruning or lengthening entries like it removes the false positive with no loss of
evasion resistance, which is strictly better than any code-side heuristic. I am happy to
withdraw this in favour of that if the list owners will take it.

🤖 Generated with Claude Code

Blocklist.censor_prompt decided whether anything had been censored by
searching its own output for the censor character, which was produced by
misc.Color.red("*"). termcolor strips ANSI escapes when stdout is not a
TTY, so in any run whose output was piped to a file the sentinel degraded
to a bare "*" and the check silently became "does the text contain an
asterisk?".

Text containing Markdown emphasis therefore blocked itself. The Cosmos3
Edge reasoner emits Markdown, so its own "**bold**" headings were reported
as blocked content with no blocklist match involved -- and the safety
verdict differed between an interactive terminal and a log file.

Measured on a 492-item Edge reasoner QA sweep: 7 outputs contained "**"
and all 7 were blocked; no output containing "**" escaped blocking. Four
further items were blocked on genuine blocklist hits. Of 16 items
describing a humanoid robot, the 15 written in plain prose passed and the
one that emitted "**humanoid robot**" was blocked.

Detect censorship with a sentinel that cannot occur in ordinary text and
keep the colourised character for display only, so the reported "Censored
Prompt" is unchanged. Adds regression tests covering Markdown emphasis,
bare asterisks, and a real blocklist hit alongside Markdown; these build
the matcher directly so they need no checkpoint download.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@rwagwani
rwagwani force-pushed the fix/guardrail-blocklist-word-join-false-positive branch from 547f048 to ed3eccb Compare August 10, 2026 15:25
@rickzw
rickzw requested a review from lfengad August 11, 2026 17:07

@lfengad lfengad left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for splitting these out — the deskin false positive is real and the read of any_next_words_form_swear_word is spot on.

My one structural concern: _has_legitimate_match re-derives matching independently of the library, so anywhere it's narrower than the library, a genuine hit gets silently let through. I found two such spots while testing — left them inline. Both are cases where stock better_profanity blocks and this branch doesn't, so they're bugs rather than part of the trade you documented.

On the trade you did document: I actually think it's reasonable for this list. It's mostly trademarks, and nobody evades a trademark filter by writing n ike — they just pick another word. So fused matching wasn't buying much here to begin with, which is also why I'd rather not pay for it with a heuristic.

Which brings me to your own closing suggestion — pruning or lengthening deskin. I think that's the better fix and the two findings below are arguments for it: any version of this heuristic re-derives the library's matching and will keep drifting from it. Could we ask the list owners first? If they'll take the data change this PR can just be withdrawn.

# apart from a match that only worked because the spaces were deleted.
self._blocklist_phrases = {
" ".join(w.lower().split()) for w in self.blocklist_words if " " in w
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a plain string set, but the library stores entries as VaryingString with CHARS_MAPPING (o->0, s->5, ...). So a leet-spelled phrase match fails this check, and then can't match the fused branch either (that entry contains a space), and we let it through.

Measured with entry boston dynamics:

text stock this branch
a boston dynamics robot blocked blocked
a b0ston dynamics robot blocked not blocked
a boston dynamic5 robot blocked not blocked

Since the list is mostly trademarks, multi-word entries are exactly the ones that matter. Storing them the way the library does makes in dispatch to VaryingString.__eq__, same as the library's own full_word in censor_words:

self._blocklist_phrases = [
    VaryingString(" ".join(w.lower().split()), char_map=self.profanity.CHARS_MAPPING)
    for w in self.blocklist_words if " " in w
]

I checked this restores both rows without re-blocking a desk in the background.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed — stored as VaryingString with CHARS_MAPPING, exactly as you wrote it. Reproduced both rows first:

'a b0ston dynamics robot'   stock=blocked  branch=NOT blocked
'a boston dynamic5 robot'   stock=blocked  branch=NOT blocked

Both block now, and a desk in the background still passes. Covered by test_leet_spelling_of_a_multi_word_entry_still_blocks.

# Phrase matches are bounded by the longest blocklist entry, but a
# separator-stripped join fuses several tokens into ONE entry word, so it
# needs its own bound. Three matches the library's own lookahead.
max_window = max(self._max_blocklist_words, _MAX_JOIN_TOKENS)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This bound is smaller than the library's actual reach. any_next_words_form_swear_word fuses up to MAX_NUMBER_COMBINATIONS + 1 tokens, and that value is monotonic — load_censor_words only ever raises it — so the module-level profanity singleton already carries ~5 over from the default wordlist before the custom list loads.

So a 4-token split escapes: entry supercalifragil, text su per cali fragil — stock blocks, this branch doesn't.

You half-caught this in the description ("passed against the production list only because that list happens to contain a 6-word phrase"). It's still coupled to that accident: one entry with a hyphen or apostrophe raises the library's reach without raising _max_blocklist_words. Taking it from the source of truth pins it:

max_window = max(self._max_blocklist_words, self.profanity.MAX_NUMBER_COMBINATIONS + 1)

(_MAX_JOIN_TOKENS on L37 then becomes unnecessary, and its "mirrors better_profanity's own lookahead" comment isn't accurate today.)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed — max_window = max(self._max_blocklist_words, self.profanity.MAX_NUMBER_COMBINATIONS + 1), and _MAX_JOIN_TOKENS is gone along with its inaccurate comment.

Worth recording that reproducing this corrected my own repro first: with every fragment marked a dictionary word, su per cali fragil now is rejected by the prose exemption rather than by the window, so the case only isolates the window when at least one part is not a dictionary word. test_join_window_follows_the_library_reach is written that way.

bl.whitelist_words = []
bl.guardrail_exempt_fused_prose = True
bl._dictionary_cache = dict(dictionary or {})
bl._max_blocklist_words = max((len(w.split()) for w in words), default=1)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worth flagging that both bugs above live in lines no test executes — the helper goes through __new__ and re-implements __init__'s bookkeeping here, so _max_blocklist_words and _blocklist_phrases are only ever exercised in their test copy.

Pulling those three assignments into a small _configure_join_bookkeeping() called from both __init__ and this helper would keep the tests pointed at the real thing. Otherwise fixing the two findings won't actually be covered.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed — the three assignments moved into _configure_join_bookkeeping(), called from both __init__ and the test helper, so the tests now exercise production's derivation.

You were right that it mattered rather than being tidiness: all three findings live in lines no test executed, and the helper's copy would have kept passing after the fixes landed.

self,
guardrail_partial_match_min_chars: int = 6,
guardrail_partial_match_letter_count: float = 0.4,
guardrail_exempt_fused_prose: bool = True,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small thing: this isn't reachable from anywhere in practice — presets.py:17 builds Blocklist() with no arguments, and that's the only real construction site. So "set it False to restore the stricter behaviour" currently means editing code. If it's meant as a genuine escape hatch it needs a config path; if not, probably worth saying so in the docstring.

Separately: flipping a guardrail's default to the laxer setting for every caller feels like a call for the guardrail owners rather than something to carry in on a false-positive fix. Might be worth looping them in regardless of which fix we land.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed the half that is mechanical: the flag now falls back to COSMOS_GUARDRAIL_EXEMPT_FUSED_PROSE when not passed explicitly, so strict matching is selectable at deploy time without editing source.

COSMOS_GUARDRAIL_EXEMPT_FUSED_PROSE=0

Your second point stands and I have not tried to settle it here — whether the relaxed setting should be the default for every caller is the guardrail owners' call. It is now called out in the PR body rather than left implicit.

@lfengad

lfengad commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

One more thing, separate from the two bugs — I want to state the net effect on strictness plainly, because I don't think it should ride in implicitly on a false-positive fix.

Even with both findings above fixed, this PR still makes the blocklist less strict by default for every caller. Three layers, only the first of which is in the description:

  1. The documented trade. A banned word split into pieces that are all multi-letter dictionary words stops counting as evasion. You call this out honestly and I think it's a reasonable trade for this list as it stands today — it's mostly trademarks, and nobody evades a trademark filter by writing n ike; they pick another word. But "reasonable given current list contents" and "correct as a permanent default" aren't the same claim. The list will change; guardrail_exempt_fused_prose=True won't.

  2. No off switch in practice. As noted on L45, presets.py:17 constructs Blocklist() with no arguments, so the strict path isn't reachable without a code edit. If we're going to relax the default, the stricter behaviour should at least be selectable from config.

  3. WordNet becomes a trust boundary. What gets through now depends on which words an external corpus happens to know. You already found it counts single letters as words and had to patch around it with the len >= 2 rule; it also happens to know ike, so that counts as ordinary prose. That's a security-relevant decision outsourced to a wordlist nobody here reviews, it drifts with corpus versions, and its failure mode is silent allow. Worth weighing even though the individual judgements are usually fine.

None of this is an argument that the false positive should stay — it's real and worth fixing. It's an argument about where. Removing deskin from the list gets the same result with zero loss of strictness and no new trust boundary, which is why I keep coming back to your own closing suggestion.

If we do land the code version instead, could we loop in whoever owns the guardrail list and config so the default change is signed off rather than inherited? Happy either way, I just don't want it to land unremarked.


whitelist = {w.lower() for w in self.whitelist_words}
raw_tokens = input_prompt.split()
tokens = [t.strip(string.punctuation).lower() for t in raw_tokens]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, one more — I went back and enumerated the gap systematically (disabling the WordNet exemption so that anything left is structural rather than the trade you intended), and this line turns up a third case.

string.punctuation and better_profanity's ALLOWED_CHARACTERS overlap on " $ ' * @ — and those are exactly the leet characters the library substitutes (a->@, o->@, s->$, and * for a/i/o/u/v/e). So a token spelled with one of them gets stripped to "" here, and then not all(window) breaks out of the window loop and the whole join branch is abandoned:

entry 'sike':
  'wear s ike shoes'  ->  stock: blocked,  this branch: blocked
  'wear $ ike shoes'  ->  stock: blocked,  this branch: NOT blocked

This one is independent of the window-size finding — it still reproduces with a production-sized window of 6 — so it'd survive fixing L209.

Stripping only the characters the library doesn't consider part of a word keeps them visible:

_STRIP_CHARS = "".join(set(string.punctuation) - ALLOWED_CHARACTERS)
...
tokens = [t.strip(_STRIP_CHARS).lower() for t in raw_tokens]

I checked that this re-blocks wear $ ike shoes without re-blocking a desk in the background.

While I was in there: a to yota car with to on the whitelist also slips through, but that one reproduces on main too — uncensor_whitelist indexes censored_words[i] positionally while the library collapses to yota into a single replacement token, so the whitelist pass wipes the marker out before your code is ever reached. Pre-existing, not yours; I'll file it separately. (It can also IndexError when a whitelisted word sits after a collapsed match, which is worth a look on its own.)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. Confirmed the overlap is exactly what you said:

set(string.punctuation) & ALLOWED_CHARACTERS  ->  " $ ' * @

Tokens are now stripped with _STRIP_CHARS = "".join(sorted(set(string.punctuation) - ALLOWED_CHARACTERS)). wear $ ike shoes blocks again, wear s ike shoes still blocks, a desk in the background still passes. Covered by test_leet_characters_are_not_stripped_from_tokens.

On the to yota case you flagged as pre-existing — it is, and it is worse than "slips through". With the production lists (whitelist is the single word flat, custom has 95 multi-word entries) the index shift also corrupts the message and can crash:

'a Snow White poster on a flat wall'  ->  'a **** poster on a flat flat'   # 'wall' overwritten
'Snow White is flat'                  ->  IndexError: list assignment index out of range
'a Snow White poster is flat'         ->  IndexError

Reproduced on main, no code changes. Happy for you to file it, or I can — say which.

root and others added 3 commits August 12, 2026 07:23
…ring

to_ascii preserves \x00 (its range is [^\x00-\x7F]), so a NUL arriving in the
input reached the sentinel check unmodified and reported the prompt as censored
with no blocklist match -- the same in-band-marker failure the sentinel replaced.

Remove it rather than substitute a space, so the sentinel cannot be used to
split a blocked word into two innocuous tokens: "n\x00ike" fuses back to a
blocked word instead of becoming "n ike".

Verified over the 492-item Edge reasoner QA corpus against the production
Cosmos-Guardrail1 word lists: identical verdicts to the previous commit on
every item.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…entry

better_profanity concatenates adjacent words with their separators removed
(utils.any_next_words_form_swear_word: full_word = cur_word + single_word)
so that a blocked word written with a space inserted mid-word is still
caught -- "n ike" for the entry "nike". The side effect is that ordinary
prose collides with short blocklist entries: "a desk in the background"
fuses into "deskin", which is on the list, and the prompt is blocked as
gore.

Note this is not governed by guardrail_partial_match_min_chars -- that
parameter applies to check_partial_match against the exact-match list,
which is a different code path.

Re-check a flagged prompt with word boundaries respected before blocking.
A match counts as legitimate when a single whitespace-delimited token
matches on its own (so joins across punctuation inside a token, such as
"desk-in", still block), when a run of tokens matches a blocklist entry
that genuinely contains spaces ("Boston Dynamics"), or when a run matches
only once the spaces are deleted AND at least one part is not ordinary
English -- the shape of a real evasion. Single letters do not count as
ordinary English, so "n ike" and "to yota" still block.

Matching is delegated to better_profanity throughout, so leet
substitutions and punctuation behave exactly as before. The re-check runs
only after something has already matched, so unflagged text costs nothing;
on a flagged 106-token output it adds ~43 ms.

_has_legitimate_match documents what the check does and does not buy: the
escape class this introduces (a banned word split into pieces that are all
multi-letter dictionary words), the escape classes that already existed
(with the entry "nike", stock code blocks "n ike" and "ni ke" but not
"nik e" or "n i k e", because the library looks ahead only two words), that
WordNet is the trust boundary and over-includes single letters, and that
the blocklist is a coarse pre-filter running alongside model based
guardrails.

Adds guardrail_exempt_fused_prose (default True). Set False to restore the
stricter previous behaviour: ordinary prose is blocked again, but no fused
match is ever let through.

Measured over the 492-item Edge reasoner QA corpus: blocks drop from 11 to
the 3 genuine trademark hits (INTEL, Nike, Huracan) with no new blocks;
strict mode adds the fused case back. Adds six regression tests covering
the fused-prose false positive, split evasions, single-letter splits,
multi-word phrases, punctuation joins, and the opt-out; they inject the
dictionary rather than reading WordNet, so they need no checkpoint and no
nltk corpus.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… behind the library

Review of NVIDIA#177 found three cases where _has_legitimate_match re-derived the
library's matching more narrowly than the library itself, so a prompt stock
better_profanity blocks was let through.

* Multi-word blocklist entries were held as plain strings, while the library
  stores them as VaryingString with CHARS_MAPPING. "b0ston dynamics" and
  "boston dynamic5" escaped. Store them the library's way so comparison
  dispatches to VaryingString.__eq__.
* The join window used a local constant of 3. MAX_NUMBER_COMBINATIONS is
  monotonic across load_censor_words calls, so the matcher's real reach is
  wider and a 4-token split escaped. Take the bound from the matcher.
* Tokens were stripped with string.punctuation, which overlaps the library's
  ALLOWED_CHARACTERS on " $ ' * @ -- the characters it substitutes for letters.
  A token spelled with one emptied and aborted the window scan, so
  "wear $ ike shoes" escaped. Strip only what the library ignores.

Also:

* Extract _configure_join_bookkeeping() and call it from both __init__ and the
  test helper, so tests exercise production's derivation rather than a copy of
  it. All three fixes above live in lines no test previously executed.
* Resolve guardrail_exempt_fused_prose from COSMOS_GUARDRAIL_EXEMPT_FUSED_PROSE
  when not passed explicitly, so strict matching is selectable without editing
  code -- presets.py constructs Blocklist() with no arguments.
* Correct the docstring's claim about the library's reach. Stock blocks "nik e"
  and "n i k e" too once a following token closes the window; the earlier claim
  was measured on fragments with no trailing word.

Verified against the 492-item Edge reasoner QA corpus with the production
Cosmos-Guardrail1 lists: 3 blocked, all genuine trademark hits, with 4_3
("sitting at a desk in the background") the only item the exemption removes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@rwagwani
rwagwani force-pushed the fix/guardrail-blocklist-word-join-false-positive branch from ed3eccb to 4953c1b Compare August 12, 2026 07:40
@rwagwani

Copy link
Copy Markdown
Collaborator Author

Thanks for the depth here — all three findings reproduced exactly as you described, and all three are fixed. Pushed as 4953c1b, rebased onto #176's new head.

finding case stock before after
phrases as plain strings a b0ston dynamics robot blocked not blocked blocked
phrases as plain strings a boston dynamic5 robot blocked not blocked blocked
join window su per cali fragil now blocked not blocked blocked
punctuation strip wear $ ike shoes blocked not blocked blocked
control — intended exemption a desk in the background blocked not blocked not blocked
control — evasion a to yota car blocked blocked blocked

Plus the bookkeeping extraction, so those lines are now under test, and an environment path for the strict setting. Tests are 18 passed. Over the 492-item QA corpus with the production lists: 3 blocked, all genuine trademark hits, with 4_3 ("sitting at a desk in the background") the only item the exemption removes.

A correction to this PR's own description, which cuts against it. While reproducing your finding about MAX_NUMBER_COMBINATIONS, I checked the "escape classes that already existed" claim and it is wrong. I measured nik e and n i k e as fragments with no trailing token; the library's window needs a following word to close. In a real sentence:

'nik e'         -> not blocked        # what I measured
'nik e shoe'    -> BLOCKED
'n i k e shoe'  -> BLOCKED

So stock is stricter than I documented, and this PR gives up more than it claimed. The four forms do all still block with the PR against the production WordNet — but only because the corpus has no synset for ke or nik, which is your trust-boundary point doing the work rather than the heuristic. Corrected in the body and the docstring rather than quietly amended.

On the strictness comment. I have not tried to resolve it in code, because I do not think it is a code question. The mechanical half is done — strict mode is now reachable via COSMOS_GUARDRAIL_EXEMPT_FUSED_PROSE=0 rather than only by editing source. Whether the relaxed setting should be the default for every caller is a decision for whoever owns this guardrail, and the correction above makes the trade look worse than when you first assessed it as reasonable. That is worth re-weighing, and I would rather you and the owners weigh it than have it inherited from this PR.

My own view has not changed from the closing paragraph: pruning or lengthening deskin gets the same result with no heuristic, no new trust boundary, and none of these five findings existing. Everything above is so that the code option is in a defensible state if the data option is refused — not an argument for preferring it.

@rwagwani

rwagwani commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator Author

Withdrawing this in favour of the data fix, as offered in the description and as you've recommended twice. Before I close it, here is the measurement that decided it — because it says the trade is considerably worse than either of us assessed, and the reasoning is worth keeping on the record.

The exposure is not confined to the trademark entries

We both reasoned from "it's mostly trademarks, and nobody evades a trademark filter by writing n ike". That premise does not hold. I split every single-word entry in the production list on WordNet and tested each 2-word split through the branch:

single-word entries:                          288
entries with a 2-dictionary-word split:       114
of those, stock blocks but this PR does not:  110

Using entries that are safe to quote here, the shape is:

batman     -> bat man          gameboy    -> game boy
chromebook -> chrome book      shotgun    -> shot gun

The concerning part is what else is in those 110. A substantial share are the content-safety entries — the explicit and graphic-violence terms the filter exists for — not the IP list. I have not quoted them for obvious reasons; the script that regenerates the list is available internally if you want to see it in full. The trademark entries are the collateral here, not the substance.

A zero-width space makes it invisible

to_ascii rewrites any non-ASCII run to a space, so a U+200B inside the word produces exactly the split the exemption then excuses, while the prompt still renders as the banned word:

renders as:      'a batman figure in the scene'
after to_ascii:  'a bat man figure in the scene'
stock = BLOCKED      this PR = not blocked

This reproduces on the content-safety entries identically. Nothing visible distinguishes the evading prompt from the plain one, which is what makes it worse than the documented "type a space" case. Introduced here — stock blocks all of them.

Narrowing the heuristic does not reach it

I prototyped capping the exemption at 2-token joins, which was the most promising way to keep the code fix. It buys back ass ass in and re-protects 33 entries whose only split is 3-way, at no cost on the 492-item corpus. But every one of the 110 escapes above is a 2-token split, so the cap changes none of them. The best available narrowing does not touch the class that matters.

What this PR was buying

One false positive, 4_3 ("sitting at a desk in the background"), out of 492 corpus items. Against a 38% evasion surface on the safety entries, reachable with an invisible character. That is not a trade worth making, and your instinct to push it back to the data was right on the first pass.

The data fix, and a wider ask

Removing deskin — or replacing it with deskinned / deskinning, which cannot fuse out of ordinary prose — clears 4_3 with no code, no strictness loss, no new trust boundary, and none of the five findings on this PR existing at all.

Worth raising with the list owners as a rule rather than one deletion, though: 114 of 288 single-word entries split into two dictionary words, so the same collision shape recurs with the next short entry added. A hygiene check at list-edit time ("does this entry fuse out of two common English words?") prevents the class. I'm happy to hand over the script that produces the list.

Also

Two things from this review that outlive the PR:

  • The uncensor_whitelist index bug you spotted in passing reproduces on main with the production lists, and it crashes rather than just slipping: 'Snow White is flat' raises IndexError: list assignment index out of range, and 'a Snow White poster on a flat wall' silently rewrites wall to flat in the user-facing message. Say the word and I'll file it with the reproduction, or it's yours if you'd rather.
  • The correction to this PR's fused-reach claim stands regardless: stock blocks nik e shoe and n i k e shoe too. The original table was measured on fragments with no trailing token.

Thanks for the depth on this one — three real bugs plus the structural objection, and the objection turned out to be the important half.

@rwagwani

Copy link
Copy Markdown
Collaborator Author

Closing per the above — withdrawn in favour of the data fix. The branch stays on the fork if the measurement or the three fixes are wanted later.

@rwagwani rwagwani closed this Aug 12, 2026
@lfengad

lfengad commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Thanks for this — and for going and measuring it instead of taking the easy exit.

Worth saying plainly: the premise you overturned was mine. I'm the one who wrote that the trade looked reasonable "for this list as it stands today," on the reasoning that it's mostly trademarks and nobody evades a trademark filter by writing n ike. You had a reviewer's blessing on that and three fixed findings; shipping was the natural ending. Instead you went and checked the assumption that was working in your favour, found 110 of 288 with the safety entries as the substance rather than the collateral, and closed your own PR. That's the right instinct and it's not a common one. The zero-width space case I hadn't considered at all.

So my structural objection was pointed the right way but argued weakly — I said the threat model didn't hold, when in fact the exposure was about an order of magnitude larger than I assumed. Your numbers, not mine.

On the two leftovers:

  • uncensor_whitelist — please do file it, and take the fix too if you're up for it. You have the reproduction against the production lists and I don't, and the IndexError plus the silent rewrite of the user-facing message are worth having written up by whoever actually measured them. I'm happy to review.
  • The list hygiene rule — I think this is the more valuable half of what you found. "114 of 288 single-word entries split into two dictionary words" is a much stronger argument to the list owners than deleting deskin one time, because it says the next short entry re-creates the same collision. Please do raise it with them and offer the script; if it helps to have a second name on that ask, add mine.

Sorry for the volume of review on something that ends up withdrawn. It was worth doing.

@rwagwani

Copy link
Copy Markdown
Collaborator Author

Closing the loop on this thread, since the follow-up work is now open elsewhere.

First, thank you for the review here and on #176. Pushing back on the exemption rather than tuning it was the right call: measured against the shipped list, the version proposed here made 108 of the 285 single-word entries evadable by inserting one space, and capping the exemption at two-token joins reached none of them. The review also turned up the invisible-character case, which is what ruled the approach out for good.

Where each of the asks landed:

  1. Withdraw this PR in favour of a data-side fix — done, and the replacement is open as fix(guardrail): let the whitelist hold phrases, so ordinary prose stops matching blocklist entries #186. Rather than removing the entry, a whitelist entry containing a space is now matched as a phrase, so exactly one spelling is exempted and every other spelling of the entry still blocks. Deleting the entry turned out to be the wrong remedy: the bare stem means the same thing as the inflected forms, so it has to keep blocking.
  2. File the whitelist restore bugfix(guardrail): whitelist restore corrupts the censored prompt, and invisible characters evade the blocklist #183, with the crash, the silent corruption, and the invisible-character folding. Your offer to review it still stands as far as I know, and it is the one blocking the other work: fix(guardrail): let the whitelist hold phrases, so ordinary prose stops matching blocklist entries #186 builds on its normalization and will be rebased down to its own two commits once fix(guardrail): whitelist restore corrupts the censored prompt, and invisible characters evade the blocklist #183 merges.
  3. Ask the list owners to act on the single entry — the ask in fix(guardrail): let the whitelist hold phrases, so ordinary prose stops matching blocklist entries #186 is to add the colliding phrase to the whitelist rather than to remove anything, so no coverage is given up.
  4. Raise the hygiene rule and offer the script — the script is in fix(guardrail): let the whitelist hold phrases, so ordinary prose stops matching blocklist entries #186 as a gate and audit mode, with the measurement you asked for: 110 of 285 single-word entries have the same collision shape, every one confirmed against the live matcher, and the audit prints entry text only behind a flag so its output can be shared safely. If the offer to co-sign the request to the list owners still stands, it would carry more weight with your name on it.
  5. Strictness changes signed off by the list owners, not inheritedfix(guardrail): let the whitelist hold phrases, so ordinary prose stops matching blocklist entries #186 changes no shipped behaviour on merge. It only gives a whitelist phrase meaning; nothing happens until the owners add one.

Reviews on #183 and #186 would be very welcome whenever you have time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants