Skip to content

test(server-test): CLI e2e tests + field-specific search examples in generated docs#916

Merged
pyramation merged 9 commits intomainfrom
devin/1774614615-cli-e2e-tests
Mar 28, 2026
Merged

test(server-test): CLI e2e tests + field-specific search examples in generated docs#916
pyramation merged 9 commits intomainfrom
devin/1774614615-cli-e2e-tests

Conversation

@pyramation
Copy link
Copy Markdown
Contributor

@pyramation pyramation commented Mar 27, 2026

Summary

Adds 11 end-to-end tests (2 suites) that exercise the full CLI pipeline: codegen → transpile → execute as child process against a running PostgreSQL database with a real GraphQL server.

Also adds field-specific search examples to all generated CLI docs so that agents and users can see concrete dot-notation flags for each search type (tsvector, trgm, BM25, pgvector, composite fullTextSearch).

CLI E2E Tests

Approach (Approach A):

  1. getConnections() spins up a real Postgres DB + GraphQL HTTP server
  2. generateCli() and generateOrm() produce TypeScript source files
  3. ts.transpileModule strips types without resolving imports (avoids needing all type packages in a temp dir)
  4. Appstash context is configured via APPSTASH_BASE_DIR pointing at a temp directory with the test server endpoint
  5. A child process (spawn, not execFileSync) runs the compiled CLI against the live server
  6. stdout is parsed and assertions verify the results

Why spawn instead of execFileSync: The GraphQL server runs in the same Node.js process. Synchronous child execution blocks the event loop, preventing the server from responding — causing a deadlock/timeout.

Suite 1 — Animals (simple-seed, 5 tests):

  1. Paginated list with --where (dot-notation) + --fields projection
  2. Cursor-based forward pagination (--limit + --after)
  3. find-first with --where.name.equalTo
  4. Combined --where + --orderBy + --fields
  5. Empty result set handling

Suite 2 — Search CLI (search-seed, 6 tests):

  1. tsvector search via --where.tsvTsv (dot-notation passthrough to server's actual filter field)
  2. trgm fuzzy matching via --where.trgmTitle.value + --where.trgmTitle.threshold
  3. Composite fullTextSearch filter via --where.fullTextSearch
  4. Search + pagination (--where.tsvTsv + --limit)
  5. pgvector error handling — verifies CLI returns {ok: false, errors: [...]} when vector arrays are passed via dot-notation (string coercion, not JSON arrays — a known CLI limitation for complex types)
  6. Schema introspection — verifies the Article type exposes expected search fields (tsvRank, titleTrgmSimilarity, bodyTrgmSimilarity, searchScore, and conditionally pgvector fields)

Package.json changes: Added @0no-co/graphql.web, gql-ast, appstash (^0.7.0), inquirerer, and nested-obj as devDependencies in server-test — these are ORM/CLI runtime deps needed by the child process.

Generated Search Examples in CLI Docs

New buildSearchExamples() and buildSearchExamplesMarkdown() helpers in docs-utils.ts. When a table has search-capable fields, generated README and skill references now include concrete examples:

  • tsvector: --where.<field> "search query" --fields title,tsvRank
  • trgm: --where.trgm<Base>.value "query" --where.trgm<Base>.threshold 0.3
  • BM25: --where.bm25<Base>.query "search query"
  • pgvector: --where.<field>.vector '[...]' --where.<field>.distance 1.0 (with CLI limitation note)
  • Composite: --where.fullTextSearch "query" (dispatches to all text adapters)
  • Combined: search + pagination + field projection example

Field-name derivation mirrors buildSearchHandler in table-command-generator.ts so examples always match the generated code. Integrated into all four generators: single-target README, single-target skills, multi-target README, and multi-target skills. Replaces the previous single generic search "query text" example.

Updates since last revision

  • Extracted shared runCli helper: Moved runCli() from inline per-suite to a shared module-level function taking (distDir, tmpHome, ...args) — eliminates ~60 lines of duplication between Suite 1 and Suite 2.
  • Template literal for runner script: Replaced [...].join('\n') array-of-strings approach with a readable RUNNER_SCRIPT template literal constant.
  • Fixed duplicate JSDoc: Removed duplicate JSDoc comment on setupAppstashContext.
  • Fixed stale header: Removed Suite 3 reference from file header comment (blueprint tests were removed in a prior revision).

Review & Testing Checklist for Human

  • buildSearchExamples field-name derivation duplicates buildSearchHandler logic: The BM25 (bodyBm25Score → bm25Body) and trgm (titleTrgmSimilarity → trgmTitle) name-mapping regexes in docs-utils.ts are copy-pasted from table-command-generator.ts. If one changes, the other will silently produce wrong examples. Consider whether this should be a shared utility.
  • Hardcoded title,tsvRank in generated examples: buildSearchExamples uses title as the example display field, which may not exist on all tables. Verify this is acceptable for example-quality docs or whether it should derive a field name from the table.
  • CLI exits 0 on GraphQL errors (Test 5): The generated CLI returns {ok: false, errors: [...]} with exit code 0 when the server rejects a query. This is the current behavior being tested, not a bug introduced by this PR — but it may be worth considering whether the CLI should exit non-zero on server errors in a follow-up.
  • buildArticlesTable() is hand-crafted (lines 609–751): Manually mirrors the search-seed schema. If the search plugin's naming conventions change or the schema.sql fixture evolves, this Table object will silently produce wrong generated code. Cross-check field names against search-seed/schema.sql and the live introspection output.
  • Hardcoded UUID in Suite 1 Test 3: Asserts node.id === 'a0000001-0000-0000-0000-000000000001' — tied to simple-seed/test-data.sql. Confirm this ID exists and won't change.

Recommended test plan: Run cd graphql/server-test && npx jest cli-e2e --verbose in CI (requires PostgreSQL) to confirm all 11 tests pass. Also verify the existing test suite still passes (pnpm test). For the doc generation changes, run cd graphql/codegen && pnpm test and inspect a generated skill reference for a search-enabled table to confirm the new examples render correctly.

Notes

  • The pnpm-lock.yaml diff is large but is purely formatting changes (single-line vs multi-line resolution objects) from a lockfile refresh — no dependency version changes beyond appstash@0.7.0.
  • The graphile-cache ERROR log during teardown (PostGraphile instance has been released) is pre-existing in other server-test suites and is harmless.
  • The companion PR for APPSTASH_BASE_DIR env var support in appstash itself was merged and published as appstash@0.7.0 via dev-utils PR #75.

Link to Devin session: https://app.devin.ai/sessions/c92c3a11450342f8875625a60fa1be28
Requested by: @pyramation


Open with Devin

Tests generated CLI commands (codegen → transpile → execute) against a
running PostgreSQL database with a real GraphQL server.

- Uses ts.transpileModule to strip types without resolving imports
- Uses async spawn (not execFileSync) to keep event loop unblocked
- Sets up appstash context pointing at test server endpoint
- Resolves NODE_PATH for pnpm's strict module isolation

5 focused corner-case tests:
1. Paginated list with --where (dot-notation) + --fields
2. Cursor-based forward pagination (--after)
3. find-first with --where.name.equalTo
4. Combined --where + --orderBy + --fields
5. Empty result set handling
@devin-ai-integration
Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

Copy link
Copy Markdown
Contributor

@devin-ai-integration devin-ai-integration bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 5 additional findings.

Open in Devin Review

…SE_DIR, bump appstash 0.7.0

- Type buildAnimalsTable() with proper Table interface from codegen
- Replace internal path.join require hacks with proper package exports
- Use APPSTASH_BASE_DIR env var instead of overriding HOME
- Bump appstash to ^0.7.0 for APPSTASH_BASE_DIR support
- Fix cliEntryPoint -> entryPoint config property
Suite 2 — Search CLI (6 tests):
- tsvector search via --where.tsvTsv (dot-notation passthrough)
- trgm fuzzy matching via --where.trgmTitle
- composite fullTextSearch filter
- search + pagination (--limit)
- pgvector similarity (conditional, skip if unavailable)
- _meta query from live server (MetaSchemaPlugin verification)

Uses search-seed fixture (5 articles with tsvector, pg_trgm, optional pgvector).
All search tests use list --where dot-notation to pass filter field names
directly to the server, testing the full pipeline: codegen -> transpile ->
spawn child process -> ORM findMany -> GraphQL -> real PostgreSQL.
- pgvector test: vector arrays can't be passed via CLI dot-notation
  (they become strings, not JSON arrays). Changed test to verify the
  CLI reports a clear GraphQL type error rather than crashing silently.
- _meta test: replaced with schema introspection test since the
  search-seed server doesn't load MetaSchemaPlugin (enableServicesApi
  is false). New test verifies Article type exposes expected search
  fields (tsvRank, titleTrgmSimilarity, bodyTrgmSimilarity, searchScore,
  and conditionally pgvector fields).
The CLI exits with code 0 even on GraphQL errors, returning
{ ok: false, errors: [...] }. Updated the test to check the
response content instead of expecting the promise to reject.
…eral for runner script

- Moved runCli() from inline per-suite to a shared module-level function
  that takes (distDir, tmpHome, ...args) — eliminates 60 lines of duplication
- Replaced string[] array joined with newlines with a readable template
  literal (RUNNER_SCRIPT constant)
- Fixed duplicate JSDoc comment on setupAppstashContext
- Removed stale Suite 3 reference from file header
When a table has search-capable fields (tsvector, trgm, BM25, pgvector),
the generated README and skill references now include concrete CLI
examples showing the exact dot-notation flags for each search type:

- tsvector:  --where.<field> "query"
- trgm:     --where.trgm<Base>.value "query" --where.trgm<Base>.threshold 0.3
- BM25:     --where.bm25<Base>.query "query"
- pgvector: --where.<field>.vector '[...]' --where.<field>.distance 1.0
- composite: --where.fullTextSearch "query"

Also adds a combined search + pagination example. Field-name derivation
mirrors buildSearchHandler so examples always match the generated code.

Integrated into all four generators: single-target README, single-target
skills, multi-target README, and multi-target skills.
@devin-ai-integration devin-ai-integration bot changed the title test(server-test): add CLI e2e tests against real PostgreSQL database test(server-test): CLI e2e tests + field-specific search examples in generated docs Mar 28, 2026
@pyramation pyramation merged commit 8c1724d into main Mar 28, 2026
44 checks passed
@pyramation pyramation deleted the devin/1774614615-cli-e2e-tests branch March 28, 2026 02:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant