Skip to content

fix(editor): skip string literals and comments in query param detection#519

Merged
debba merged 2 commits into
TabularisDB:mainfrom
ymadd:fix/issue-458-query-params
Jul 24, 2026
Merged

fix(editor): skip string literals and comments in query param detection#519
debba merged 2 commits into
TabularisDB:mainfrom
ymadd:fix/issue-458-query-params

Conversation

@ymadd

@ymadd ymadd commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Fixes #458

Problem

extractQueryParams / interpolateQueryParams ran a plain regex over the whole SQL text, so :y in

SELECT * FROM categories WHERE value = 'x:y'

popped the Query Parameters modal — and filling in a value silently rewrote the inside of the string literal, corrupting the query. Anything with colons in data hit this: URLs, timestamps, key:value pairs, JSON stored as text.

Approach

Instead of adding a second scanner, this reuses the dialect-aware tokenizer that already powers the statement splitter (#225):

  • scanNonCodeSpans(sql, dialect?) (new export from src/utils/sqlSplitter) returns the spans of every string literal and comment, driving scanToken with the same state updates as the splitter. tokenizer.ts internals stay private; the collector is an iterative worklist (see Hardening below).
  • queryParameters.ts runs detection and interpolation on a length-preserving mask (non-code spans blanked with spaces) with the existing regex unchanged, so the ::cast exclusion behaves exactly as before. Interpolation is position-based over the original text (mask indices transfer 1:1), so values can never land inside a literal — including after astral characters like '😀:id', which the old full-text replace corrupted.
  • Editor.tsx threads the driver's sql_dialect capability (introduced in feat(sql): first-party splitter + per-driver dialect #225) into all six call sites, and the Params-button probe is memoized since it runs in the render path on every keystroke.

Fallback semantics. With an undefined/unknown dialect nothing is masked: behavior is byte-for-byte the old one (the optional dialect argument keeps both functions backward compatible; the pre-existing tests pass unmodified). Guessing a dialect is measurably worse than not masking — e.g. generic quote rules read MySQL's 'a\'b' as an unterminated string and would swallow every real parameter after it. Masking with the correct dialect only ever removes false positives; masking with a wrong one can hide real parameters.

MySQL executable comments (/*!50700 ... */) are rescanned rather than masked, so :params in the executable payload stay live while literals inside it are still protected. MariaDB's /*M! ... */ gets the same treatment at the masking layer only — the splitter still folds it as a comment, so MySQL proper never receives a comment-only statement.

Hardening

  • The executable-comment rescan is an iterative worklist with a depth cap instead of native recursion: a pasted ~15KB chain of /*! markers used to overflow the call stack (reachable from the render path on MySQL connections). Past the cap the tail is left unmasked — the same fail-safe direction as the unknown-dialect fallback.
  • Dialect lookup uses Object.hasOwn, so prototype member names ('__proto__', 'toString') fall back cleanly instead of handing a non-DialectOptions value to the tokenizer. normalizeDialect had the same latent issue and got the same fix.
  • Interpolation ignores parameter names that only resolve through Object.prototype (:toString no longer splices function source into the SQL).

Dialect-preset corrections

Building the masking layer surfaced preset gaps that also affect statement splitting; each is fixed with splitter canary tests:

Dialect Fix Reference
MySQL/MariaDB # line comments recognized (any position, no whitespace requirement). Previously a ; inside # note; mis-split the script. MySQL comment syntax
MSSQL "double-quoted" identifiers added to the quote rules (identifiers under default QUOTED_IDENTIFIER ON, strings when OFF — non-code either way). SET QUOTED_IDENTIFIER
MSSQL Block comments now nest, per T-SQL semantics. Slash-star comments
PostgreSQL A dollar quote glued to an identifier (foo$tag$) is no longer treated as a string opener. Lexical structure
MariaDB /*M! ... */ payloads rescanned at the masking layer (see above). Comment syntax

Known limitations (follow-up)

  • A dedicated MariaDB dialect: tokenizer-level /*M! statement semantics (executing them as statements is only correct for MariaDB, and both servers currently share the mysql dialect).
  • NO_BACKSLASH_ESCAPES: the sql_mode is per-session server state a static lexer cannot know; under that mode 'a\' is a closed literal. Handling it would require reading @@sql_mode through the driver.

I'll open a follow-up issue for these after this lands.

Tests

59 new cases: 28 in tests/utils/sqlSplitter/nonCodeSpans.test.ts (span exactness and sorted/non-overlapping/in-bounds invariants, UTF-16 indices with astral chars, unterminated strings/comments, dialect fallback incl. prototype-key names, per-dialect matrix, DELIMITER collision cases, GO/slash advancement, the depth-cap boundary, and a regression test for the 15KB /*! chain), 27 in tests/utils/queryParameters.test.ts (issue repro, boundary cases like 'x'::text / /*c*/:id / :na/*c*/me, maskless-fallback equivalence, position-based interpolation incl. unknown-param preservation), and 4 splitter canaries in tests/utils/sqlSplitter/dialects.test.ts.

Full suite: 3170 passed, tsc --noEmit clean, eslint clean. The pre-existing queryParameters tests pass unmodified.

A plain regex over the whole SQL text treated `:y` in `WHERE v = 'x:y'`
as a bind parameter, popping the Query Parameters modal and corrupting
the literal on interpolation (TabularisDB#458).

Detection and interpolation now run on a length-preserving mask built
from a new sqlSplitter `scanNonCodeSpans()` export, which reuses the
dialect-aware tokenizer to locate string literals and comments
(dollar quotes, E-strings, q-quotes, bracket identifiers, nested
comments, and MySQL executable-comment payloads included). Replacement
is position-based on the original text so values never land inside a
literal. Without a known dialect nothing is masked and the previous
behavior is preserved verbatim, because lexing with the wrong quote
rules could swallow real parameters (e.g. generic rules on MySQL's
backslash escapes). The editor threads the active driver dialect into
all call sites and memoizes the Params-button probe that runs in the
render path.

Hardening from adversarial review: the executable-comment rescan is an
iterative worklist with a depth cap instead of native recursion (a
pasted ~15KB chain of `/*!` markers used to overflow the call stack
from the render path), dialect lookup uses Object.hasOwn so prototype
member names like '__proto__' fall back cleanly instead of crashing
the tokenizer (normalizeDialect had the same latent issue), and
interpolation ignores parameter names that only resolve through
Object.prototype instead of splicing function source into the SQL.

Dialect-preset corrections surfaced by the same review (these also fix
statement splitting, with canary tests): MySQL `#` line comments are
now recognized, MSSQL gains double-quoted identifiers and nested block
comments per T-SQL semantics, PostgreSQL dollar quotes glued to an
identifier (`foo$tag$`) are no longer treated as string openers, and
MariaDB `/*M! ... */` conditional comments are rescanned at the
masking layer only (the splitter still folds them as comments so MySQL
proper never receives a comment-only statement).

Known limitations left for follow-up: a dedicated MariaDB dialect
(tokenizer-level `/*M!` statement semantics) and the session-dependent
NO_BACKSLASH_ESCAPES sql_mode, which a static lexer cannot know.
@ymadd
ymadd force-pushed the fix/issue-458-query-params branch from c92f585 to fb8ace1 Compare July 23, 2026 18:09
@debba

debba commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

Looks perfect to me.
Merging it.

@debba
debba merged commit f702aff into TabularisDB:main Jul 24, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Query params modal triggers on ":" inside string literals

2 participants