fix(editor): skip string literals and comments in query param detection#519
Merged
Conversation
A plain regex over the whole SQL text treated `:y` in `WHERE v = 'x:y'` as a bind parameter, popping the Query Parameters modal and corrupting the literal on interpolation (TabularisDB#458). Detection and interpolation now run on a length-preserving mask built from a new sqlSplitter `scanNonCodeSpans()` export, which reuses the dialect-aware tokenizer to locate string literals and comments (dollar quotes, E-strings, q-quotes, bracket identifiers, nested comments, and MySQL executable-comment payloads included). Replacement is position-based on the original text so values never land inside a literal. Without a known dialect nothing is masked and the previous behavior is preserved verbatim, because lexing with the wrong quote rules could swallow real parameters (e.g. generic rules on MySQL's backslash escapes). The editor threads the active driver dialect into all call sites and memoizes the Params-button probe that runs in the render path. Hardening from adversarial review: the executable-comment rescan is an iterative worklist with a depth cap instead of native recursion (a pasted ~15KB chain of `/*!` markers used to overflow the call stack from the render path), dialect lookup uses Object.hasOwn so prototype member names like '__proto__' fall back cleanly instead of crashing the tokenizer (normalizeDialect had the same latent issue), and interpolation ignores parameter names that only resolve through Object.prototype instead of splicing function source into the SQL. Dialect-preset corrections surfaced by the same review (these also fix statement splitting, with canary tests): MySQL `#` line comments are now recognized, MSSQL gains double-quoted identifiers and nested block comments per T-SQL semantics, PostgreSQL dollar quotes glued to an identifier (`foo$tag$`) are no longer treated as string openers, and MariaDB `/*M! ... */` conditional comments are rescanned at the masking layer only (the splitter still folds them as comments so MySQL proper never receives a comment-only statement). Known limitations left for follow-up: a dedicated MariaDB dialect (tokenizer-level `/*M!` statement semantics) and the session-dependent NO_BACKSLASH_ESCAPES sql_mode, which a static lexer cannot know.
ymadd
force-pushed
the
fix/issue-458-query-params
branch
from
July 23, 2026 18:09
c92f585 to
fb8ace1
Compare
Collaborator
|
Looks perfect to me. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #458
Problem
extractQueryParams/interpolateQueryParamsran a plain regex over the whole SQL text, so:yinpopped the Query Parameters modal — and filling in a value silently rewrote the inside of the string literal, corrupting the query. Anything with colons in data hit this: URLs, timestamps,
key:valuepairs, JSON stored as text.Approach
Instead of adding a second scanner, this reuses the dialect-aware tokenizer that already powers the statement splitter (#225):
scanNonCodeSpans(sql, dialect?)(new export fromsrc/utils/sqlSplitter) returns the spans of every string literal and comment, drivingscanTokenwith the same state updates as the splitter.tokenizer.tsinternals stay private; the collector is an iterative worklist (see Hardening below).queryParameters.tsruns detection and interpolation on a length-preserving mask (non-code spans blanked with spaces) with the existing regex unchanged, so the::castexclusion behaves exactly as before. Interpolation is position-based over the original text (mask indices transfer 1:1), so values can never land inside a literal — including after astral characters like'😀:id', which the old full-text replace corrupted.Editor.tsxthreads the driver'ssql_dialectcapability (introduced in feat(sql): first-party splitter + per-driver dialect #225) into all six call sites, and the Params-button probe is memoized since it runs in the render path on every keystroke.Fallback semantics. With an
undefined/unknown dialect nothing is masked: behavior is byte-for-byte the old one (the optionaldialectargument keeps both functions backward compatible; the pre-existing tests pass unmodified). Guessing a dialect is measurably worse than not masking — e.g. generic quote rules read MySQL's'a\'b'as an unterminated string and would swallow every real parameter after it. Masking with the correct dialect only ever removes false positives; masking with a wrong one can hide real parameters.MySQL executable comments (
/*!50700 ... */) are rescanned rather than masked, so:paramsin the executable payload stay live while literals inside it are still protected. MariaDB's/*M! ... */gets the same treatment at the masking layer only — the splitter still folds it as a comment, so MySQL proper never receives a comment-only statement.Hardening
/*!markers used to overflow the call stack (reachable from the render path on MySQL connections). Past the cap the tail is left unmasked — the same fail-safe direction as the unknown-dialect fallback.Object.hasOwn, so prototype member names ('__proto__','toString') fall back cleanly instead of handing a non-DialectOptionsvalue to the tokenizer.normalizeDialecthad the same latent issue and got the same fix.Object.prototype(:toStringno longer splices function source into the SQL).Dialect-preset corrections
Building the masking layer surfaced preset gaps that also affect statement splitting; each is fixed with splitter canary tests:
#line comments recognized (any position, no whitespace requirement). Previously a;inside# note;mis-split the script."double-quoted"identifiers added to the quote rules (identifiers under defaultQUOTED_IDENTIFIER ON, strings whenOFF— non-code either way).foo$tag$) is no longer treated as a string opener./*M! ... */payloads rescanned at the masking layer (see above).Known limitations (follow-up)
/*M!statement semantics (executing them as statements is only correct for MariaDB, and both servers currently share themysqldialect).NO_BACKSLASH_ESCAPES: the sql_mode is per-session server state a static lexer cannot know; under that mode'a\'is a closed literal. Handling it would require reading@@sql_modethrough the driver.I'll open a follow-up issue for these after this lands.
Tests
59 new cases: 28 in
tests/utils/sqlSplitter/nonCodeSpans.test.ts(span exactness and sorted/non-overlapping/in-bounds invariants, UTF-16 indices with astral chars, unterminated strings/comments, dialect fallback incl. prototype-key names, per-dialect matrix,DELIMITERcollision cases,GO/slash advancement, the depth-cap boundary, and a regression test for the 15KB/*!chain), 27 intests/utils/queryParameters.test.ts(issue repro, boundary cases like'x'::text//*c*/:id/:na/*c*/me, maskless-fallback equivalence, position-based interpolation incl. unknown-param preservation), and 4 splitter canaries intests/utils/sqlSplitter/dialects.test.ts.Full suite: 3170 passed,
tsc --noEmitclean,eslintclean. The pre-existingqueryParameterstests pass unmodified.