From 53a15543eedbc904c87e3d7f4bde7eae1e47053e Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 20 Aug 2026 11:36:18 -0700 Subject: [PATCH 01/35] docs(plan): ai search optimization plan for threadplane.ai Co-Authored-By: Claude Opus 5 --- .../2026-08-20-ai-search-optimization.md | 2174 +++++++++++++++++ 1 file changed, 2174 insertions(+) create mode 100644 docs/superpowers/plans/2026-08-20-ai-search-optimization.md diff --git a/docs/superpowers/plans/2026-08-20-ai-search-optimization.md b/docs/superpowers/plans/2026-08-20-ai-search-optimization.md new file mode 100644 index 000000000..3d56fc8d9 --- /dev/null +++ b/docs/superpowers/plans/2026-08-20-ai-search-optimization.md @@ -0,0 +1,2174 @@ +# AI Search Optimization (threadplane.ai) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make threadplane.ai maximally eligible for, and measurable in, Google's generative AI search surfaces (AI Overviews, AI Mode, Discover) plus third-party AI answer engines — by fixing the technical-structure gaps Google's AI optimization guide actually calls out, and by standing up a Search Console API analysis harness for ongoing diagnosis. + +**Architecture:** Four phases. Phase 1 builds a dependency-free Google Search Console API harness (service-account JWT via `node:crypto` + `fetch`) that pulls Search Analytics, Sitemaps, and URL Inspection data into committed JSON snapshots plus a derived report. Phase 2 fixes the technical-structure gaps found in the audit (no JSON-LD anywhere, no sitemap `lastmod`, no article dates/authors in metadata, one shared OG image, brand-name inconsistency, heading text polluted by anchor glyphs, agent-hostile markup). Phase 3 closes content and E-E-A-T gaps. Phase 4 wires ongoing measurement (AI-crawler and AI-referral tracking through the existing PostHog server client, plus a documented manual export for the Search Console Generative AI report, which is **not** in any API as of 2026-08). + +**Tech Stack:** Next.js App Router (`apps/website`), TypeScript, vitest (`apps/website/vite.config.mts`), `tsx` for scripts (already a root devDependency), `posthog-node` (already an `apps/website` dependency), Google Search Console API v3 over raw `fetch`. **No new npm dependencies** — deliberately, because regenerating `package-lock.json` on macOS drops the Linux `@next/swc-*` bindings and breaks CI. + +--- + +## Background: What the audit found + +Evidence gathered 2026-08-20 against the live site and `apps/website` on this branch. + +**Already good — do not "fix" these:** +- `robots.txt` is fully permissive (`User-agent: * / Allow: /`) and points at the sitemap. All AI crawlers (Google-Extended, GPTBot, ClaudeBot, PerplexityBot) are therefore allowed. Correct posture for AI visibility. +- Sitemap is generated from real route data: 140 URLs (11 static, 3 solutions, ~117 docs, 9 blog). +- Pages are server-rendered with full content in the HTML (home 1,675 words / docs installation 1,985 / blog tutorial 3,103). No JS-SEO problem. +- Canonicals are emitted on every page that goes through `createPageMetadata`. +- Newer blog posts already use question-form H2s (`How do we bind Angular to it?`) — exactly the shape RAG systems extract well. +- All 21 homepage `` tags have `alt`. +- `llms.txt` / `llms-full.txt` exist. Google explicitly says these do **nothing** for Google Search — keep them anyway for non-Google engines, but do not invest further there. + +**Gaps this plan fixes:** + +| # | Gap | Evidence | +|---|-----|----------| +| G1 | Zero structured data sitewide | `grep -rl "application/ld+json" src lib` → no matches; live HTML `ld+json` count = 0 on home, docs, and blog | +| G2 | Sitemap has no `lastmod` | `src/app/sitemap.ts` emits only `changeFrequency`/`priority` — the two signals Google ignores, and omits the one it uses | +| G3 | Blog metadata has no `publishedTime`, `modifiedTime`, `authors`, or `tags` | `createPageMetadata` in `src/lib/site-metadata.ts` has no article fields | +| G4 | Every page shares one OG image (`/opengraph-image`) | `DEFAULT_SOCIAL_IMAGE` is the only value passed | +| G5 | Brand name inconsistency in titles | blog titles use `— ThreadPlane`, docs use `- Threadplane`, root uses `Threadplane` | +| G6 | Heading text is polluted by the anchor glyph | live docs H2s extract as `#Prerequisites`; `MdxRenderer.tsx:37,43` renders a literal `#` text node *before* the heading text | +| G7 | Blog posts contain zero images/diagrams | live blog HTML: `img` count = 0. Google's guide explicitly asks for high-quality images/video | +| G8 | No author/about surface for E-E-A-T | `blogAuthors` has a bio string but there is no `/about` page and no `Person` entity anywhere | +| G9 | No visibility measurement for AI surfaces | no crawler logging, no AI-referral tracking, no GSC harness | +| G10 | 3 `solutions/*` pages are programmatic and thin | `src/lib/solutions-data.ts` is 9.4 KB total for 3 pages — scaled-content-abuse adjacent if expanded | + +**What Google's guide says NOT to do (do not let scope creep add these):** +- No `llms.txt`-style AI-specific files as a *Search* tactic. +- No content "chunking" for retrieval — multi-topic pages are understood fine. +- No AI-specific keyword rewrites; write naturally. +- No pursuit of inauthentic mentions. +- Structured data is *optional* for AI features (it gates rich results). We add it in Phase 2 because it is cheap, it is a rich-result and entity-clarity win, and it costs nothing at runtime — not because it unlocks AI Overviews. + +**Deliberately out of scope:** the guide's third pillar — Merchant Center product feeds, Google Business Profile, and the Business Agent / Universal Commerce Protocol surfaces — is aimed at retail and local businesses. Threadplane sells developer licenses through Stripe with no physical location and no product catalog, so none of it applies. This is a scope decision, not an oversight. + +**Critical constraint on measurement:** as of 2026-08-11, the Search Console **Generative AI performance report is UI-only**. `searchanalytics.query`'s `type` field still accepts only `web|image|video|news|discover|googleNews`; there is no `aiMode`/`aiOverview` type, no `searchAppearance` value for AI features, and no BigQuery export. Any plan or tool that claims to pull AI Overview impressions programmatically is wrong. Phase 1 therefore uses the API for everything it *can* answer (query/page performance, striking-distance, index coverage), and Phase 4 documents a manual CSV export for the AI report. + +--- + +## File Structure + +**Phase 1 — GSC harness (new, self-contained):** +- `apps/website/scripts/gsc/auth.ts` — service-account JWT → access token. Only file that touches crypto. +- `apps/website/scripts/gsc/api.ts` — thin typed wrappers over the four GSC endpoints. +- `apps/website/scripts/gsc/pull.ts` — CLI entrypoint: writes raw snapshots to `apps/website/.gsc/`. +- `apps/website/scripts/gsc/report.ts` — CLI entrypoint: reads snapshots, writes a markdown report. Pure functions, unit-testable. +- `apps/website/scripts/gsc/analysis.ts` — pure analysis functions (no I/O) consumed by `report.ts`. +- `apps/website/scripts/gsc/analysis.spec.ts` — vitest coverage of the analysis functions. +- `apps/website/scripts/gsc/README.md` — service-account setup runbook. + +**Phase 2 — site metadata and structured data:** +- `apps/website/src/lib/structured-data.ts` (new) — JSON-LD builders, pure functions. +- `apps/website/src/lib/structured-data.spec.ts` (new). +- `apps/website/src/components/shared/JsonLd.tsx` (new) — one render component. +- `apps/website/src/lib/site-metadata.ts` (modify) — article fields, brand constant, `getSitemapEntries`. +- `apps/website/src/app/sitemap.ts` (modify) — emit `lastModified`. +- `apps/website/src/app/layout.tsx` (modify) — Organization + WebSite JSON-LD. +- `apps/website/src/app/blog/[slug]/page.tsx` (modify) — article metadata + Article/Breadcrumb JSON-LD. +- `apps/website/src/app/blog/[slug]/opengraph-image.tsx` (new) — per-post OG image. +- `apps/website/src/app/docs/[library]/[section]/[slug]/page.tsx` (modify) — TechArticle/Breadcrumb JSON-LD. +- `apps/website/src/components/docs/MdxRenderer.tsx` (modify) — anchor glyph out of heading text. + +**Phase 3 — content:** +- `apps/website/src/app/about/page.tsx` (new). +- `apps/website/content/blog/*.mdx` (modify — add diagrams). + +**Phase 4 — measurement:** +- `apps/website/src/middleware.ts` (new) — AI crawler + AI referral capture. +- `apps/website/src/lib/analytics/ai-traffic.ts` (new) — pure UA/referrer classification. +- `apps/website/src/lib/analytics/ai-traffic.spec.ts` (new). +- `apps/website/src/lib/analytics/events.ts` (modify) — register the two new events. +- `docs/gtm/ai-search-measurement.md` (new) — the manual GSC AI-report runbook. + +--- + +## Phase 1 — Search Console API analysis harness + +### Task 1: Service-account auth + +**Files:** +- Create: `apps/website/scripts/gsc/auth.ts` +- Create: `apps/website/scripts/gsc/README.md` + +- [ ] **Step 1: Write the setup runbook** + +Create `apps/website/scripts/gsc/README.md`: + +```markdown +# Search Console API harness + +## One-time setup + +1. In Google Cloud console, create (or reuse) a project and enable the + **Google Search Console API** (`searchconsole.googleapis.com`). +2. Create a service account. No project-level IAM roles are needed. +3. Create a JSON key for that service account and download it. +4. In Search Console, open the `threadplane.ai` **Domain property** → + Settings → Users and permissions → Add user → paste the service + account's `client_email` → permission **Full** (required: the URL + Inspection API rejects "Restricted" users). +5. Export the key for local use — do NOT commit it: + + export GSC_SERVICE_ACCOUNT_JSON="$(cat ~/secrets/threadplane-gsc.json)" + export GSC_SITE_URL="sc-domain:threadplane.ai" + +## Usage + + npx tsx apps/website/scripts/gsc/pull.ts # writes apps/website/.gsc/*.json + npx tsx apps/website/scripts/gsc/report.ts # writes apps/website/.gsc/report.md + +## What this CANNOT do + +The Search Console **Generative AI performance report** (AI Overviews / +AI Mode impressions and clicks) is UI-only as of 2026-08. It is not in +`searchanalytics.query`, not in `searchAppearance`, and not in the +BigQuery bulk export. See `docs/gtm/ai-search-measurement.md` for the +manual export procedure. +``` + +- [ ] **Step 2: Write the auth module** + +Create `apps/website/scripts/gsc/auth.ts`: + +```ts +// SPDX-License-Identifier: MIT +import { createSign } from 'node:crypto'; + +const TOKEN_URL = 'https://oauth2.googleapis.com/token'; +const SCOPE = 'https://www.googleapis.com/auth/webmasters.readonly'; + +interface ServiceAccountKey { + client_email: string; + private_key: string; +} + +function base64url(value: object): string { + return Buffer.from(JSON.stringify(value)).toString('base64url'); +} + +export function readServiceAccountKey(): ServiceAccountKey { + const raw = process.env['GSC_SERVICE_ACCOUNT_JSON']; + if (!raw) { + throw new Error( + 'GSC_SERVICE_ACCOUNT_JSON is not set. See apps/website/scripts/gsc/README.md.', + ); + } + const parsed = JSON.parse(raw) as Partial; + if (!parsed.client_email || !parsed.private_key) { + throw new Error('GSC_SERVICE_ACCOUNT_JSON is missing client_email or private_key.'); + } + return { client_email: parsed.client_email, private_key: parsed.private_key }; +} + +export async function getAccessToken(nowSeconds = Math.floor(Date.now() / 1000)): Promise { + const key = readServiceAccountKey(); + const signingInput = [ + base64url({ alg: 'RS256', typ: 'JWT' }), + base64url({ + iss: key.client_email, + scope: SCOPE, + aud: TOKEN_URL, + iat: nowSeconds, + exp: nowSeconds + 3600, + }), + ].join('.'); + const signature = createSign('RSA-SHA256') + .update(signingInput) + .sign(key.private_key, 'base64url'); + + const response = await fetch(TOKEN_URL, { + method: 'POST', + headers: { 'content-type': 'application/x-www-form-urlencoded' }, + body: new URLSearchParams({ + grant_type: 'urn:ietf:params:oauth:grant-type:jwt-bearer', + assertion: `${signingInput}.${signature}`, + }), + }); + + if (!response.ok) { + throw new Error(`Token exchange failed: ${response.status} ${await response.text()}`); + } + const json = (await response.json()) as { access_token?: string }; + if (!json.access_token) throw new Error('Token exchange returned no access_token.'); + return json.access_token; +} +``` + +- [ ] **Step 3: Verify auth end to end** + +Run (with the env vars from the README exported): + +```bash +npx tsx -e "import('./apps/website/scripts/gsc/auth.ts').then(async m => console.log((await m.getAccessToken()).slice(0, 12) + '…'))" +``` + +Expected: a token prefix like `ya29.c.b49…`. If it returns `401 invalid_grant`, the key is wrong; if the later API calls return `403`, the service account was not added to the Search Console property. + +- [ ] **Step 4: Ignore the snapshot directory** + +Append to `apps/website/.gitignore` (create the file if absent): + +``` +.gsc/ +``` + +- [ ] **Step 5: Commit** + +```bash +git add apps/website/scripts/gsc/auth.ts apps/website/scripts/gsc/README.md apps/website/.gitignore +git commit -m "feat(website): search console api service-account auth" +``` + +--- + +### Task 2: Typed API wrappers + +**Files:** +- Create: `apps/website/scripts/gsc/api.ts` + +- [ ] **Step 1: Write the API module** + +Create `apps/website/scripts/gsc/api.ts`: + +```ts +// SPDX-License-Identifier: MIT +import { getAccessToken } from './auth'; + +const BASE = 'https://www.googleapis.com/webmasters/v3'; +const INSPECT_URL = 'https://searchconsole.googleapis.com/v1/urlInspection/index:inspect'; + +export type Dimension = 'query' | 'page' | 'country' | 'device' | 'date' | 'searchAppearance'; + +export interface SearchAnalyticsRow { + keys: string[]; + clicks: number; + impressions: number; + ctr: number; + position: number; +} + +export function getSiteUrl(): string { + return process.env['GSC_SITE_URL'] ?? 'sc-domain:threadplane.ai'; +} + +async function authedFetch(url: string, init: RequestInit & { token: string }): Promise { + const { token, ...rest } = init; + const response = await fetch(url, { + ...rest, + headers: { ...(rest.headers ?? {}), authorization: `Bearer ${token}`, 'content-type': 'application/json' }, + }); + if (!response.ok) { + throw new Error(`${url} → ${response.status} ${await response.text()}`); + } + return response.json(); +} + +/** + * Search Analytics. NOTE: `type` accepts only web|image|video|news|discover| + * googleNews. There is no AI Overviews / AI Mode type as of 2026-08. + */ +export async function querySearchAnalytics(options: { + startDate: string; + endDate: string; + dimensions: Dimension[]; + rowLimit?: number; + startRow?: number; + type?: 'web' | 'image' | 'video' | 'news' | 'discover' | 'googleNews'; +}): Promise { + const token = await getAccessToken(); + const site = encodeURIComponent(getSiteUrl()); + const rows: SearchAnalyticsRow[] = []; + let startRow = options.startRow ?? 0; + const rowLimit = options.rowLimit ?? 25000; + + for (;;) { + const page = (await authedFetch(`${BASE}/sites/${site}/searchAnalytics/query`, { + token, + method: 'POST', + body: JSON.stringify({ + startDate: options.startDate, + endDate: options.endDate, + dimensions: options.dimensions, + type: options.type ?? 'web', + rowLimit, + startRow, + dataState: 'all', + }), + })) as { rows?: SearchAnalyticsRow[] }; + const batch = page.rows ?? []; + rows.push(...batch); + if (batch.length < rowLimit) break; + startRow += rowLimit; + } + return rows; +} + +export async function listSitemaps(): Promise { + const token = await getAccessToken(); + return authedFetch(`${BASE}/sites/${encodeURIComponent(getSiteUrl())}/sitemaps`, { + token, + method: 'GET', + }); +} + +export interface InspectionResult { + url: string; + verdict: string; + coverageState: string; + lastCrawlTime: string | null; + robotsTxtState: string; + indexingState: string; + googleCanonical: string | null; + userCanonical: string | null; +} + +export async function inspectUrl(inspectionUrl: string): Promise { + const token = await getAccessToken(); + const raw = (await authedFetch(INSPECT_URL, { + token, + method: 'POST', + body: JSON.stringify({ inspectionUrl, siteUrl: getSiteUrl(), languageCode: 'en-US' }), + })) as { + inspectionResult?: { + indexStatusResult?: Record; + }; + }; + const status = raw.inspectionResult?.indexStatusResult ?? {}; + return { + url: inspectionUrl, + verdict: status['verdict'] ?? 'UNKNOWN', + coverageState: status['coverageState'] ?? 'UNKNOWN', + lastCrawlTime: status['lastCrawlTime'] ?? null, + robotsTxtState: status['robotsTxtState'] ?? 'UNKNOWN', + indexingState: status['indexingState'] ?? 'UNKNOWN', + googleCanonical: status['googleCanonical'] ?? null, + userCanonical: status['userCanonical'] ?? null, + }; +} +``` + +- [ ] **Step 2: Verify against the live property** + +```bash +npx tsx -e "import('./apps/website/scripts/gsc/api.ts').then(async m => console.log(JSON.stringify(await m.listSitemaps(), null, 2)))" +``` + +Expected: JSON listing `https://threadplane.ai/sitemap.xml` with `lastSubmitted` and `contents`. A `403 User does not have sufficient permission` means step 4 of the README was skipped. + +- [ ] **Step 3: Commit** + +```bash +git add apps/website/scripts/gsc/api.ts +git commit -m "feat(website): typed search console api wrappers" +``` + +--- + +### Task 3: Snapshot puller + +**Files:** +- Create: `apps/website/scripts/gsc/pull.ts` + +- [ ] **Step 1: Write the puller** + +Create `apps/website/scripts/gsc/pull.ts`: + +```ts +// SPDX-License-Identifier: MIT +import fs from 'node:fs'; +import path from 'node:path'; +import { inspectUrl, listSitemaps, querySearchAnalytics } from './api'; + +const OUT_DIR = path.join(process.cwd(), 'apps', 'website', '.gsc'); + +function isoDaysAgo(days: number): string { + const date = new Date(Date.now() - days * 86_400_000); + return date.toISOString().slice(0, 10); +} + +function write(name: string, value: unknown): void { + fs.mkdirSync(OUT_DIR, { recursive: true }); + fs.writeFileSync(path.join(OUT_DIR, name), JSON.stringify(value, null, 2)); + console.log(`wrote .gsc/${name}`); +} + +async function sitemapUrls(): Promise { + const response = await fetch('https://threadplane.ai/sitemap.xml'); + const xml = await response.text(); + return [...xml.matchAll(/([^<]+)<\/loc>/g)].map((m) => m[1]); +} + +async function main(): Promise { + // Search Analytics data lags ~2 days; end 3 days back for a stable window. + const endDate = isoDaysAgo(3); + const startDate = isoDaysAgo(93); + + write('meta.json', { startDate, endDate, pulledAt: new Date().toISOString() }); + write('queries.json', await querySearchAnalytics({ startDate, endDate, dimensions: ['query'] })); + write('pages.json', await querySearchAnalytics({ startDate, endDate, dimensions: ['page'] })); + write( + 'query-page.json', + await querySearchAnalytics({ startDate, endDate, dimensions: ['query', 'page'] }), + ); + write('dates.json', await querySearchAnalytics({ startDate, endDate, dimensions: ['date'] })); + write( + 'discover.json', + await querySearchAnalytics({ startDate, endDate, dimensions: ['page'], type: 'discover' }), + ); + write('sitemaps.json', await listSitemaps()); + + // URL Inspection is quota-limited (2000/day, 600/min). Serialize with a small delay. + const urls = await sitemapUrls(); + const inspections = []; + for (const url of urls) { + inspections.push(await inspectUrl(url)); + await new Promise((resolve) => setTimeout(resolve, 150)); + } + write('inspections.json', inspections); +} + +main().catch((error: unknown) => { + console.error(error); + process.exitCode = 1; +}); +``` + +- [ ] **Step 2: Run the pull** + +```bash +npx tsx apps/website/scripts/gsc/pull.ts +``` + +Expected: eight `wrote .gsc/*.json` lines. The inspection loop takes ~30s for 140 URLs. If `discover.json` comes back empty, that is normal — Discover only reports once the property has Discover impressions. + +- [ ] **Step 3: Commit** + +```bash +git add apps/website/scripts/gsc/pull.ts +git commit -m "feat(website): search console snapshot puller" +``` + +--- + +### Task 4: Analysis + report + +**Files:** +- Create: `apps/website/scripts/gsc/analysis.ts` +- Create: `apps/website/scripts/gsc/analysis.spec.ts` +- Create: `apps/website/scripts/gsc/report.ts` +- Modify: `apps/website/vite.config.mts` + +- [ ] **Step 1: Widen the vitest include so scripts are tested** + +In `apps/website/vite.config.mts`, change: + +```ts + include: ['src/**/*.spec.ts', 'src/**/*.spec.tsx'], +``` + +to: + +```ts + include: ['src/**/*.spec.ts', 'src/**/*.spec.tsx', 'scripts/**/*.spec.ts'], +``` + +- [ ] **Step 2: Write the failing test** + +Create `apps/website/scripts/gsc/analysis.spec.ts`: + +```ts +import { describe, expect, it } from 'vitest'; +import { findStrikingDistance, findUnindexed, findZeroImpressionPages } from './analysis'; + +const rows = [ + { keys: ['angular langgraph chat'], clicks: 0, impressions: 400, ctr: 0, position: 11.2 }, + { keys: ['threadplane'], clicks: 90, impressions: 100, ctr: 0.9, position: 1.1 }, + { keys: ['obscure long tail'], clicks: 0, impressions: 3, ctr: 0, position: 42 }, +]; + +describe('findStrikingDistance', () => { + it('returns rows ranking 5-20 with meaningful impressions, best opportunity first', () => { + const result = findStrikingDistance(rows, { minImpressions: 50 }); + expect(result.map((r) => r.keys[0])).toEqual(['angular langgraph chat']); + }); +}); + +describe('findZeroImpressionPages', () => { + it('lists sitemap URLs that earned no impressions in the window', () => { + const result = findZeroImpressionPages( + ['https://threadplane.ai/a', 'https://threadplane.ai/b'], + [{ keys: ['https://threadplane.ai/a'], clicks: 1, impressions: 10, ctr: 0.1, position: 5 }], + ); + expect(result).toEqual(['https://threadplane.ai/b']); + }); +}); + +describe('findUnindexed', () => { + it('flags inspections whose verdict is not PASS', () => { + const result = findUnindexed([ + { url: 'https://threadplane.ai/a', verdict: 'PASS', coverageState: 'Submitted and indexed', lastCrawlTime: null, robotsTxtState: 'ALLOWED', indexingState: 'INDEXING_ALLOWED', googleCanonical: null, userCanonical: null }, + { url: 'https://threadplane.ai/b', verdict: 'NEUTRAL', coverageState: 'Discovered - currently not indexed', lastCrawlTime: null, robotsTxtState: 'ALLOWED', indexingState: 'INDEXING_ALLOWED', googleCanonical: null, userCanonical: null }, + ]); + expect(result.map((r) => r.url)).toEqual(['https://threadplane.ai/b']); + }); +}); +``` + +- [ ] **Step 3: Run it and confirm it fails** + +```bash +npx nx test website +``` + +Expected: FAIL — `Failed to resolve import "./analysis"`. + +- [ ] **Step 4: Write the analysis module** + +Create `apps/website/scripts/gsc/analysis.ts`: + +```ts +// SPDX-License-Identifier: MIT +import type { InspectionResult, SearchAnalyticsRow } from './api'; + +/** Queries ranking just off page one — the cheapest ranking wins available. */ +export function findStrikingDistance( + rows: SearchAnalyticsRow[], + options: { minImpressions: number }, +): SearchAnalyticsRow[] { + return rows + .filter( + (row) => + row.position >= 5 && + row.position <= 20 && + row.impressions >= options.minImpressions, + ) + .sort((a, b) => b.impressions - a.impressions); +} + +/** Sitemap URLs Google never showed for anything in the window. */ +export function findZeroImpressionPages( + sitemapUrls: string[], + pageRows: SearchAnalyticsRow[], +): string[] { + const seen = new Set(pageRows.map((row) => row.keys[0].replace(/\/$/, ''))); + return sitemapUrls.filter((url) => !seen.has(url.replace(/\/$/, ''))); +} + +/** Inspections that are not cleanly indexed. */ +export function findUnindexed(inspections: InspectionResult[]): InspectionResult[] { + return inspections.filter((inspection) => inspection.verdict !== 'PASS'); +} + +/** Pages Google canonicalized somewhere other than where we asked — duplicate-content smell. */ +export function findCanonicalMismatches(inspections: InspectionResult[]): InspectionResult[] { + return inspections.filter( + (inspection) => + inspection.googleCanonical !== null && + inspection.userCanonical !== null && + inspection.googleCanonical !== inspection.userCanonical, + ); +} + +/** Queries with strong impressions but a CTR well below the position-typical rate. */ +export function findWeakCtr( + rows: SearchAnalyticsRow[], + options: { minImpressions: number; maxCtr: number }, +): SearchAnalyticsRow[] { + return rows + .filter( + (row) => + row.impressions >= options.minImpressions && + row.position <= 10 && + row.ctr < options.maxCtr, + ) + .sort((a, b) => b.impressions - a.impressions); +} +``` + +- [ ] **Step 5: Run the tests and confirm they pass** + +```bash +npx nx test website +``` + +Expected: PASS. + +- [ ] **Step 6: Write the report generator** + +Create `apps/website/scripts/gsc/report.ts`: + +```ts +// SPDX-License-Identifier: MIT +import fs from 'node:fs'; +import path from 'node:path'; +import type { InspectionResult, SearchAnalyticsRow } from './api'; +import { + findCanonicalMismatches, + findStrikingDistance, + findUnindexed, + findWeakCtr, + findZeroImpressionPages, +} from './analysis'; + +const DIR = path.join(process.cwd(), 'apps', 'website', '.gsc'); + +function read(name: string): T { + return JSON.parse(fs.readFileSync(path.join(DIR, name), 'utf8')) as T; +} + +function table(rows: SearchAnalyticsRow[], headers: string[], limit = 30): string { + const head = `| ${headers.join(' | ')} |\n| ${headers.map(() => '---').join(' | ')} |`; + const body = rows + .slice(0, limit) + .map( + (row) => + `| ${row.keys.join(' | ')} | ${row.clicks} | ${row.impressions} | ${(row.ctr * 100).toFixed(1)}% | ${row.position.toFixed(1)} |`, + ) + .join('\n'); + return `${head}\n${body}`; +} + +function main(): void { + const meta = read<{ startDate: string; endDate: string }>('meta.json'); + const queries = read('queries.json'); + const pages = read('pages.json'); + const inspections = read('inspections.json'); + const sitemapUrls = inspections.map((inspection) => inspection.url); + + const totals = queries.reduce( + (acc, row) => ({ clicks: acc.clicks + row.clicks, impressions: acc.impressions + row.impressions }), + { clicks: 0, impressions: 0 }, + ); + + const unindexed = findUnindexed(inspections); + const mismatches = findCanonicalMismatches(inspections); + const orphans = findZeroImpressionPages(sitemapUrls, pages); + + const report = [ + `# threadplane.ai — Search Console report`, + ``, + `Window: ${meta.startDate} → ${meta.endDate}. Total clicks ${totals.clicks}, impressions ${totals.impressions}.`, + ``, + `> Google's AI Overviews / AI Mode impressions are NOT included — that report is UI-only.`, + `> See docs/gtm/ai-search-measurement.md.`, + ``, + `## Index health`, + ``, + `- Sitemap URLs inspected: ${inspections.length}`, + `- Not cleanly indexed: ${unindexed.length}`, + `- Google canonical ≠ our canonical: ${mismatches.length}`, + `- Zero-impression pages in window: ${orphans.length}`, + ``, + `### Not indexed`, + ``, + unindexed.length + ? unindexed.map((i) => `- ${i.url} — ${i.coverageState}`).join('\n') + : '_none_', + ``, + `### Canonical mismatches`, + ``, + mismatches.length + ? mismatches.map((i) => `- ${i.url} → Google chose ${i.googleCanonical}`).join('\n') + : '_none_', + ``, + `### Zero-impression pages`, + ``, + orphans.length ? orphans.map((url) => `- ${url}`).join('\n') : '_none_', + ``, + `## Striking distance (position 5–20, ≥50 impressions)`, + ``, + table(findStrikingDistance(queries, { minImpressions: 50 }), ['Query', 'Clicks', 'Impr', 'CTR', 'Pos']), + ``, + `## Weak CTR on page one (≥100 impressions, CTR < 2%)`, + ``, + `Title/description rewrite candidates.`, + ``, + table(findWeakCtr(queries, { minImpressions: 100, maxCtr: 0.02 }), ['Query', 'Clicks', 'Impr', 'CTR', 'Pos']), + ``, + `## Top pages`, + ``, + table(pages.slice(0, 30), ['Page', 'Clicks', 'Impr', 'CTR', 'Pos']), + ``, + ].join('\n'); + + fs.writeFileSync(path.join(DIR, 'report.md'), report); + console.log('wrote .gsc/report.md'); +} + +main(); +``` + +- [ ] **Step 7: Generate and read the report** + +```bash +npx tsx apps/website/scripts/gsc/report.ts && cat apps/website/.gsc/report.md +``` + +Expected: `wrote .gsc/report.md`, followed by the rendered report. **Read it before continuing** — the "Not indexed", "Canonical mismatches", and "Zero-impression pages" sections determine whether extra remediation tasks are needed beyond Phase 2. + +- [ ] **Step 8: Add npm scripts** + +In the root `package.json` `scripts` block, add: + +```json + "gsc:pull": "npx tsx apps/website/scripts/gsc/pull.ts", + "gsc:report": "npx tsx apps/website/scripts/gsc/report.ts", +``` + +- [ ] **Step 9: Commit** + +```bash +git add apps/website/scripts/gsc/analysis.ts apps/website/scripts/gsc/analysis.spec.ts apps/website/scripts/gsc/report.ts apps/website/vite.config.mts package.json +git commit -m "feat(website): search console analysis report" +``` + +--- + +## Phase 2 — Technical structure + +### Task 5: Sitemap `lastmod` + +Google ignores `changefreq` and `priority` entirely but does use `lastmod` when it is honest. Blog posts have real dates; docs and static routes use the source file's mtime. + +**Files:** +- Modify: `apps/website/src/lib/site-metadata.ts` +- Modify: `apps/website/src/app/sitemap.ts` +- Modify: `apps/website/src/lib/site-metadata.spec.ts` + +- [ ] **Step 1: Write the failing test** + +Append to `apps/website/src/lib/site-metadata.spec.ts`: + +```ts +describe('getSitemapEntries', () => { + it('emits a lastModified date for every route', async () => { + const { getSitemapEntries } = await import('./site-metadata'); + const entries = getSitemapEntries(); + expect(entries.length).toBeGreaterThan(100); + for (const entry of entries) { + expect(entry.lastModified).toBeInstanceOf(Date); + expect(Number.isNaN(entry.lastModified.getTime())).toBe(false); + } + }); + + it('uses the post date as lastModified for blog routes', async () => { + const { getSitemapEntries } = await import('./site-metadata'); + const entry = getSitemapEntries().find((e) => e.route === '/blog/angular-chat-app-tutorial-with-ag-ui'); + expect(entry?.lastModified.toISOString().slice(0, 10)).toBe('2026-08-13'); + }); +}); +``` + +- [ ] **Step 2: Run it and confirm it fails** + +```bash +npx nx test website +``` + +Expected: FAIL — `getSitemapEntries is not a function`. + +- [ ] **Step 3: Implement `getSitemapEntries`** + +In `apps/website/src/lib/site-metadata.ts`, add these imports at the top: + +```ts +import fs from 'node:fs'; +import path from 'node:path'; +``` + +and append: + +```ts +export interface SitemapEntry { + route: string; + lastModified: Date; +} + +function fileModifiedTime(relativePath: string): Date { + const candidates = [ + path.join(process.cwd(), 'apps', 'website', relativePath), + path.join(process.cwd(), relativePath), + ]; + for (const candidate of candidates) { + if (fs.existsSync(candidate)) return fs.statSync(candidate).mtime; + } + return new Date(); +} + +export function getSitemapEntries(): SitemapEntry[] { + const blogDates = new Map(getAllPosts().map((post) => [`/blog/${post.slug}`, new Date(post.frontmatter.date)])); + + return getSitemapRoutes().map((route) => { + const blogDate = blogDates.get(route); + if (blogDate) return { route, lastModified: blogDate }; + + if (route.startsWith('/docs/')) { + const [, , library, section, slug] = route.split('/'); + return { + route, + lastModified: fileModifiedTime(path.join('content', 'docs', library, section, `${slug}.mdx`)), + }; + } + + return { route, lastModified: fileModifiedTime(path.join('src', 'app', route === '/' ? 'page.tsx' : `${route.replace(/^\//, '')}/page.tsx`)) }; + }); +} +``` + +- [ ] **Step 4: Rewrite the sitemap route** + +Replace the body of `apps/website/src/app/sitemap.ts` with: + +```ts +import type { MetadataRoute } from 'next'; +import { getCanonicalUrl, getSitemapEntries } from '../lib/site-metadata'; + +export default function sitemap(): MetadataRoute.Sitemap { + return getSitemapEntries().map((entry) => ({ + url: getCanonicalUrl(entry.route), + lastModified: entry.lastModified, + })); +} +``` + +- [ ] **Step 5: Run the tests and confirm they pass** + +```bash +npx nx test website +``` + +Expected: PASS. + +- [ ] **Step 6: Verify the built sitemap** + +```bash +npx nx build website && grep -m3 -A3 "" dist/apps/website/.next/server/app/sitemap.xml.body 2>/dev/null || npx nx serve website +``` + +If the build artifact path differs, serve locally and `curl -s http://localhost:3000/sitemap.xml | head -20`. Expected: each `` now carries a `` and no longer carries ``/``. + +- [ ] **Step 7: Commit** + +```bash +git add apps/website/src/lib/site-metadata.ts apps/website/src/lib/site-metadata.spec.ts apps/website/src/app/sitemap.ts +git commit -m "feat(website): emit honest lastmod in sitemap" +``` + +--- + +### Task 6: Article metadata + brand-name consistency + +`ThreadPlane` vs `Threadplane` splits the brand entity; `publishedTime`/`author` are freshness and E-E-A-T signals that AI surfaces attribute with. + +**Files:** +- Modify: `apps/website/src/lib/site-metadata.ts` +- Modify: `apps/website/src/lib/site-metadata.spec.ts` +- Modify: `apps/website/src/app/blog/[slug]/page.tsx` +- Modify: `apps/website/src/lib/docs.ts` + +- [ ] **Step 1: Write the failing test** + +Append to `apps/website/src/lib/site-metadata.spec.ts`: + +```ts +describe('createPageMetadata article fields', () => { + it('emits openGraph article dates, authors, and tags', () => { + const metadata = createPageMetadata({ + title: 'Post — Threadplane', + description: 'A post.', + pathname: '/blog/post', + type: 'article', + article: { + publishedTime: '2026-08-13', + modifiedTime: '2026-08-14', + authors: ['Brian Love'], + tags: ['angular', 'ag-ui'], + }, + }); + const openGraph = metadata.openGraph as Record; + expect(openGraph['publishedTime']).toBe('2026-08-13'); + expect(openGraph['modifiedTime']).toBe('2026-08-14'); + expect(openGraph['authors']).toEqual(['Brian Love']); + expect(openGraph['tags']).toEqual(['angular', 'ag-ui']); + }); + + it('accepts a page-specific social image', () => { + const metadata = createPageMetadata({ + title: 'Post — Threadplane', + description: 'A post.', + pathname: '/blog/post', + image: '/blog/post/opengraph-image', + }); + const openGraph = metadata.openGraph as { images: string[] }; + expect(openGraph.images).toEqual(['/blog/post/opengraph-image']); + }); +}); + +describe('brand name', () => { + it('uses one canonical spelling', () => { + expect(SITE_NAME).toBe('Threadplane'); + }); +}); +``` + +Add `SITE_NAME` to the existing import list at the top of that spec file. + +- [ ] **Step 2: Run it and confirm it fails** + +```bash +npx nx test website +``` + +Expected: FAIL — `publishedTime` is `undefined`. + +- [ ] **Step 3: Extend `createPageMetadata`** + +In `apps/website/src/lib/site-metadata.ts`, replace the `createPageMetadata` function with: + +```ts +export interface ArticleMetadata { + publishedTime: string; + modifiedTime?: string; + authors?: string[]; + tags?: string[]; +} + +export function createPageMetadata({ + title, + description, + pathname, + type = 'article', + image = DEFAULT_SOCIAL_IMAGE, + article, +}: { + title: string; + description: string; + pathname: string; + type?: 'article' | 'website'; + image?: string; + article?: ArticleMetadata; +}): Metadata { + const canonicalPath = getCanonicalPath(pathname); + + return { + title, + description, + alternates: { canonical: canonicalPath }, + openGraph: { + title, + description, + url: canonicalPath, + siteName: SITE_NAME, + type, + images: [image], + ...(article + ? { + publishedTime: article.publishedTime, + modifiedTime: article.modifiedTime ?? article.publishedTime, + authors: article.authors, + tags: article.tags, + } + : {}), + }, + twitter: { + card: 'summary_large_image', + title, + description, + images: [image], + }, + }; +} +``` + +- [ ] **Step 4: Wire the blog route** + +In `apps/website/src/app/blog/[slug]/page.tsx`, replace the body of `generateMetadata` with: + +```ts +export async function generateMetadata({ params }: Params): Promise { + const { slug } = await params; + const post = getPostBySlug(slug); + if (!post || post.frontmatter.draft) { + return { title: 'Post not found — Threadplane' }; + } + const author = getAuthor(post.frontmatter.author); + return createPageMetadata({ + title: `${post.frontmatter.title} — Threadplane`, + description: post.frontmatter.description, + pathname: `/blog/${post.slug}`, + type: 'article', + image: `/blog/${post.slug}/opengraph-image`, + article: { + publishedTime: post.frontmatter.date, + authors: [author.name], + tags: post.frontmatter.tags, + }, + }); +} +``` + +- [ ] **Step 5: Unify the docs title separator** + +In `apps/website/src/lib/docs.ts:89`, change: + +```ts + const title = `${doc.title} - ${libraryTitle} Docs - Threadplane`; +``` + +to: + +```ts + const title = `${doc.title} — ${libraryTitle} Docs — Threadplane`; +``` + +- [ ] **Step 6: Sweep the remaining mis-cased brand names** + +```bash +grep -rn "ThreadPlane" apps/website/src apps/website/content | grep -v node_modules +``` + +Fix every hit to `Threadplane` (the brand is one word, capital T only). Then confirm: + +```bash +grep -rn "ThreadPlane" apps/website/src apps/website/content | grep -v node_modules | wc -l +``` + +Expected: `0`. + +- [ ] **Step 7: Run the tests and confirm they pass** + +```bash +npx nx test website +``` + +Expected: PASS. + +- [ ] **Step 8: Commit** + +```bash +git add apps/website/src apps/website/content +git commit -m "feat(website): article metadata + canonical brand spelling" +``` + +--- + +### Task 7: JSON-LD structured data + +**Files:** +- Create: `apps/website/src/lib/structured-data.ts` +- Create: `apps/website/src/lib/structured-data.spec.ts` +- Create: `apps/website/src/components/shared/JsonLd.tsx` + +- [ ] **Step 1: Write the failing test** + +Create `apps/website/src/lib/structured-data.spec.ts`: + +```ts +import { describe, expect, it } from 'vitest'; +import { + breadcrumbJsonLd, + organizationJsonLd, + softwareSourceCodeJsonLd, + techArticleJsonLd, + websiteJsonLd, + blogPostingJsonLd, +} from './structured-data'; + +describe('organizationJsonLd', () => { + it('describes Threadplane with an absolute url and logo', () => { + const data = organizationJsonLd(); + expect(data['@type']).toBe('Organization'); + expect(data['name']).toBe('Threadplane'); + expect(String(data['url'])).toBe('https://threadplane.ai/'); + }); +}); + +describe('websiteJsonLd', () => { + it('is a WebSite node pointing at the origin', () => { + expect(websiteJsonLd()['@type']).toBe('WebSite'); + }); +}); + +describe('blogPostingJsonLd', () => { + it('carries headline, dates, author, and absolute urls', () => { + const data = blogPostingJsonLd({ + title: 'A Post', + description: 'About things.', + slug: 'a-post', + datePublished: '2026-08-13', + authorName: 'Brian Love', + tags: ['angular'], + }); + expect(data['@type']).toBe('BlogPosting'); + expect(data['headline']).toBe('A Post'); + expect(data['datePublished']).toBe('2026-08-13'); + expect(data['dateModified']).toBe('2026-08-13'); + expect((data['author'] as Record)['name']).toBe('Brian Love'); + expect(String(data['url'])).toBe('https://threadplane.ai/blog/a-post'); + }); +}); + +describe('techArticleJsonLd', () => { + it('describes a docs page', () => { + const data = techArticleJsonLd({ + title: 'Installation', + description: 'Install it.', + pathname: '/docs/chat/getting-started/installation', + }); + expect(data['@type']).toBe('TechArticle'); + expect(String(data['url'])).toBe('https://threadplane.ai/docs/chat/getting-started/installation'); + }); +}); + +describe('breadcrumbJsonLd', () => { + it('numbers positions from 1 and resolves absolute urls', () => { + const data = breadcrumbJsonLd([ + { name: 'Docs', pathname: '/docs' }, + { name: 'Chat', pathname: '/docs/chat' }, + ]); + const items = data['itemListElement'] as Record[]; + expect(items).toHaveLength(2); + expect(items[0]['position']).toBe(1); + expect(String(items[1]['item'])).toBe('https://threadplane.ai/docs/chat'); + }); +}); + +describe('softwareSourceCodeJsonLd', () => { + it('marks Threadplane as an Angular TypeScript library', () => { + const data = softwareSourceCodeJsonLd(); + expect(data['@type']).toBe('SoftwareSourceCode'); + expect(data['programmingLanguage']).toBe('TypeScript'); + }); +}); +``` + +- [ ] **Step 2: Run it and confirm it fails** + +```bash +npx nx test website +``` + +Expected: FAIL — `Failed to resolve import "./structured-data"`. + +- [ ] **Step 3: Implement the builders** + +Create `apps/website/src/lib/structured-data.ts`: + +```ts +// SPDX-License-Identifier: MIT +import { getCanonicalUrl, SITE_NAME } from './site-metadata'; + +export type JsonLdNode = Record; + +const ORGANIZATION_ID = getCanonicalUrl('/') + '#organization'; + +export function organizationJsonLd(): JsonLdNode { + return { + '@context': 'https://schema.org', + '@type': 'Organization', + '@id': ORGANIZATION_ID, + name: SITE_NAME, + url: getCanonicalUrl('/'), + logo: getCanonicalUrl('/logos/threadplane-mark.svg'), + description: + 'Threadplane builds the Angular UI layer for production agent applications on LangGraph and AG-UI-compatible runtimes.', + sameAs: ['https://github.com/blove/angular-agent-framework', 'https://www.npmjs.com/org/threadplane'], + }; +} + +export function websiteJsonLd(): JsonLdNode { + return { + '@context': 'https://schema.org', + '@type': 'WebSite', + '@id': getCanonicalUrl('/') + '#website', + name: SITE_NAME, + url: getCanonicalUrl('/'), + publisher: { '@id': ORGANIZATION_ID }, + }; +} + +export function softwareSourceCodeJsonLd(): JsonLdNode { + return { + '@context': 'https://schema.org', + '@type': 'SoftwareSourceCode', + name: '@threadplane/chat', + description: + 'Signal-native Angular chat UI primitives bound to a runtime-neutral Agent contract, with adapters for LangGraph and AG-UI.', + programmingLanguage: 'TypeScript', + runtimePlatform: 'Angular', + codeRepository: 'https://github.com/blove/angular-agent-framework', + author: { '@id': ORGANIZATION_ID }, + license: 'https://threadplane.ai/docs/licensing', + }; +} + +export function blogPostingJsonLd(post: { + title: string; + description: string; + slug: string; + datePublished: string; + dateModified?: string; + authorName: string; + tags?: string[]; +}): JsonLdNode { + return { + '@context': 'https://schema.org', + '@type': 'BlogPosting', + headline: post.title, + description: post.description, + url: getCanonicalUrl(`/blog/${post.slug}`), + mainEntityOfPage: getCanonicalUrl(`/blog/${post.slug}`), + datePublished: post.datePublished, + dateModified: post.dateModified ?? post.datePublished, + image: getCanonicalUrl(`/blog/${post.slug}/opengraph-image`), + keywords: post.tags, + author: { '@type': 'Person', name: post.authorName, url: getCanonicalUrl('/about') }, + publisher: { '@id': ORGANIZATION_ID }, + }; +} + +export function techArticleJsonLd(doc: { + title: string; + description: string; + pathname: string; + dateModified?: string; +}): JsonLdNode { + return { + '@context': 'https://schema.org', + '@type': 'TechArticle', + headline: doc.title, + description: doc.description, + url: getCanonicalUrl(doc.pathname), + mainEntityOfPage: getCanonicalUrl(doc.pathname), + ...(doc.dateModified ? { dateModified: doc.dateModified } : {}), + author: { '@id': ORGANIZATION_ID }, + publisher: { '@id': ORGANIZATION_ID }, + proficiencyLevel: 'Expert', + }; +} + +export function breadcrumbJsonLd(crumbs: { name: string; pathname: string }[]): JsonLdNode { + return { + '@context': 'https://schema.org', + '@type': 'BreadcrumbList', + itemListElement: crumbs.map((crumb, index) => ({ + '@type': 'ListItem', + position: index + 1, + name: crumb.name, + item: getCanonicalUrl(crumb.pathname), + })), + }; +} + +export function faqJsonLd(entries: { question: string; answer: string }[]): JsonLdNode { + return { + '@context': 'https://schema.org', + '@type': 'FAQPage', + mainEntity: entries.map((entry) => ({ + '@type': 'Question', + name: entry.question, + acceptedAnswer: { '@type': 'Answer', text: entry.answer }, + })), + }; +} +``` + +- [ ] **Step 4: Run the tests and confirm they pass** + +```bash +npx nx test website +``` + +Expected: PASS. If `organizationJsonLd` fails on the logo path, run `ls apps/website/public/logos` and use the actual filename. + +- [ ] **Step 5: Write the render component** + +Create `apps/website/src/components/shared/JsonLd.tsx`: + +```tsx +// SPDX-License-Identifier: MIT +import type { JsonLdNode } from '../../lib/structured-data'; + +/** + * Renders schema.org JSON-LD. Content is generated from our own data, never + * from user input, so `dangerouslySetInnerHTML` is safe here; `<` is still + * escaped to keep a stray value from closing the script tag. + */ +export function JsonLd({ data }: { data: JsonLdNode | JsonLdNode[] }) { + const json = JSON.stringify(data).replace(/; +} +``` + +- [ ] **Step 6: Commit** + +```bash +git add apps/website/src/lib/structured-data.ts apps/website/src/lib/structured-data.spec.ts apps/website/src/components/shared/JsonLd.tsx +git commit -m "feat(website): schema.org json-ld builders" +``` + +--- + +### Task 8: Mount JSON-LD on layout, blog, and docs + +**Files:** +- Modify: `apps/website/src/app/layout.tsx` +- Modify: `apps/website/src/app/blog/[slug]/page.tsx` +- Modify: `apps/website/src/app/docs/[library]/[section]/[slug]/page.tsx` + +- [ ] **Step 1: Add sitewide entity nodes** + +In `apps/website/src/app/layout.tsx`, add imports: + +```tsx +import { JsonLd } from '../components/shared/JsonLd'; +import { organizationJsonLd, softwareSourceCodeJsonLd, websiteJsonLd } from '../lib/structured-data'; +``` + +and inside ``, as the first child: + +```tsx + +``` + +- [ ] **Step 2: Add BlogPosting + breadcrumbs to blog posts** + +In `apps/website/src/app/blog/[slug]/page.tsx`, add imports: + +```tsx +import { JsonLd } from '../../../components/shared/JsonLd'; +import { blogPostingJsonLd, breadcrumbJsonLd } from '../../../lib/structured-data'; +``` + +and inside `BlogPostPage`, as the first child of the outermost `
`: + +```tsx + +``` + +- [ ] **Step 3: Add TechArticle + breadcrumbs to docs pages** + +In `apps/website/src/app/docs/[library]/[section]/[slug]/page.tsx`, add imports: + +```tsx +import { JsonLd } from '../../../../../components/shared/JsonLd'; +import { breadcrumbJsonLd, techArticleJsonLd } from '../../../../../lib/structured-data'; +``` + +and inside `DocsPage`, after the `if (!doc) notFound();` guard, as the first child of the returned outermost `
`: + +```tsx + +``` + +If `doc` has no `description` field, use `getDocDescription(doc.content, libConfig.description)` and import it from `../../../../../lib/docs`. + +- [ ] **Step 4: Build and verify the emitted JSON-LD** + +```bash +npx nx build website +``` + +Expected: build succeeds. Then serve and check: + +```bash +npx nx serve website +``` + +In a second shell: + +```bash +curl -s http://localhost:3000/blog/angular-chat-app-tutorial-with-ag-ui | grep -c 'application/ld+json' +``` + +Expected: `2` (layout node array + page node array). Then validate the payload: + +```bash +curl -s http://localhost:3000/blog/angular-chat-app-tutorial-with-ag-ui | python3 -c "import sys,re,json;[json.loads(m) for m in re.findall(r'ld\+json\"[^>]*>(.*?)', sys.stdin.read(), re.S)] and print('valid json-ld')" +``` + +Expected: `valid json-ld`. + +- [ ] **Step 5: Validate against Google** + +Paste the deployed URL into https://search.google.com/test/rich-results after the PR merges and confirm zero errors. Record the result in the PR description. + +- [ ] **Step 6: Commit** + +```bash +git add apps/website/src/app +git commit -m "feat(website): mount json-ld on layout, blog, and docs" +``` + +--- + +### Task 9: Per-post OG images + +**Files:** +- Create: `apps/website/src/app/blog/[slug]/opengraph-image.tsx` + +- [ ] **Step 1: Read the existing generator for style parity** + +```bash +cat apps/website/src/app/opengraph-image.tsx +``` + +Match its `size`, `contentType`, font loading, and color usage in the next step rather than inventing a new look. + +- [ ] **Step 2: Write the per-post generator** + +Create `apps/website/src/app/blog/[slug]/opengraph-image.tsx`: + +```tsx +// SPDX-License-Identifier: MIT +import { ImageResponse } from 'next/og'; +import { getAllPosts, getPostBySlug } from '../../../lib/blog'; +import { getAuthor } from '../../../lib/blog-authors'; + +export const size = { width: 1200, height: 630 }; +export const contentType = 'image/png'; +export const alt = 'Threadplane blog post'; + +export function generateStaticParams() { + return getAllPosts().map((post) => ({ slug: post.slug })); +} + +export default async function OpengraphImage({ params }: { params: Promise<{ slug: string }> }) { + const { slug } = await params; + const post = getPostBySlug(slug); + const title = post?.frontmatter.title ?? 'Threadplane'; + const author = post ? getAuthor(post.frontmatter.author).name : 'Threadplane'; + + return new ImageResponse( + ( +
+
+ Threadplane · Blog +
+
{title}
+
{author} · threadplane.ai
+
+ ), + size, + ); +} +``` + +- [ ] **Step 3: Verify the image renders** + +```bash +npx nx serve website +``` + +In a second shell: + +```bash +curl -s -o /tmp/og.png -w "%{http_code} %{content_type} %{size_download}\n" "http://localhost:3000/blog/angular-chat-app-tutorial-with-ag-ui/opengraph-image" +``` + +Expected: `200 image/png` with a size over 10000 bytes. Open `/tmp/og.png` and confirm the title is not clipped for the longest post title (`Build Fullstack Agentic Angular Apps Using AG-UI`). + +- [ ] **Step 4: Commit** + +```bash +git add "apps/website/src/app/blog/[slug]/opengraph-image.tsx" +git commit -m "feat(website): per-post opengraph images" +``` + +--- + +### Task 10: Clean heading text + agent-friendly markup + +Right now every docs H2 extracts as `#Prerequisites`. That glyph rides along into snippets, TOCs, and any model reading the DOM. Google's agent-friendly guidance is the same checklist as accessibility: semantic elements, meaningful link text, clean headings. + +**Files:** +- Modify: `apps/website/src/components/docs/MdxRenderer.tsx` +- Modify: `apps/website/src/app/global.css` + +- [ ] **Step 1: Confirm the defect on the live site** + +```bash +curl -s https://threadplane.ai/docs/chat/getting-started/installation | grep -o ']*>.\{0,60\}' | head -3 +``` + +Expected: heading markup where the `#` anchor precedes the text. + +- [ ] **Step 2: Move the anchor after the text and hide it from assistive tech and extraction** + +In `apps/website/src/components/docs/MdxRenderer.tsx`, at both line 37 and line 43, change the anchor so it renders **after** the heading children and carries `aria-hidden`: + +```tsx + {id ? ( + + ) : null} +``` + +Ensure the `{children}` expression precedes this block in both heading components. + +- [ ] **Step 3: Keep the anchor reachable without polluting the text** + +Append to `apps/website/src/app/global.css`: + +```css +.heading-anchor { + margin-left: 0.35rem; + opacity: 0; + text-decoration: none; + transition: opacity 120ms ease-in-out; +} + +h1:hover > .heading-anchor, +h2:hover > .heading-anchor, +h3:hover > .heading-anchor, +h4:hover > .heading-anchor { + opacity: 0.5; +} +``` + +- [ ] **Step 4: Verify the extracted heading text is clean** + +```bash +npx nx serve website +``` + +In a second shell: + +```bash +curl -s http://localhost:3000/docs/chat/getting-started/installation | python3 -c " +import sys, re +html = sys.stdin.read() +for level, text in re.findall(r']*>(.*?)', html, re.S)[:8]: + print(level, re.sub(r'<[^>]+>', '', text).strip()) +" +``` + +Expected: `2 Prerequisites`, `2 1. Install the packages`, … with no leading `#`. + +- [ ] **Step 5: Commit** + +```bash +git add apps/website/src/components/docs/MdxRenderer.tsx apps/website/src/app/global.css +git commit -m "fix(website): keep anchor glyphs out of heading text" +``` + +--- + +## Phase 3 — Content and E-E-A-T + +### Task 11: `/about` page with a Person entity + +Google's guidance leans on first-hand expertise and people-first content. There is currently no page on the site that establishes who writes it. AI answer engines cite named, attributable authors far more readily than anonymous docs. + +**Files:** +- Create: `apps/website/src/app/about/page.tsx` +- Modify: `apps/website/src/lib/site-metadata.ts` +- Modify: `apps/website/src/components/shared/Footer.tsx` + +- [ ] **Step 1: Create the page** + +Create `apps/website/src/app/about/page.tsx`: + +```tsx +// SPDX-License-Identifier: MIT +import type { Metadata } from 'next'; +import { JsonLd } from '../../components/shared/JsonLd'; +import { createPageMetadata, getCanonicalUrl } from '../../lib/site-metadata'; +import { blogAuthors } from '../../lib/blog-authors'; + +export const metadata: Metadata = createPageMetadata({ + title: 'About — Threadplane', + description: + 'Threadplane is built by Brian Love, an Angular consultant and open-source maintainer, to give Angular teams a production UI layer for LangGraph and AG-UI agents.', + pathname: '/about', + type: 'website', +}); + +export default function AboutPage() { + const brian = blogAuthors['brian']; + + return ( +
+ +

About Threadplane

+

+ Threadplane is the Angular UI layer for production agent applications. It exists because + Angular teams shipping LangGraph and AG-UI agents kept rebuilding the same last mile: + streaming, durable threads, interrupts, tool calls, and generative UI. +

+

Who builds it

+

+ {brian.name} — {brian.role}. {brian.bio} +

+

What we will not do

+

+ No runtime lock-in and no abandoned majors. Threadplane binds to a runtime-neutral Agent + contract, and every adapter ships against the same conformance suite. +

+
+ ); +} +``` + +Replace the prose with Brian's own words before merging — this page is an E-E-A-T signal, and boilerplate defeats the purpose. Do not fabricate biography details. + +- [ ] **Step 2: Add it to the sitemap** + +In `apps/website/src/lib/site-metadata.ts`, add `'/about'` to the `staticRoutes` array in `getSitemapRoutes`, after `'/pricing'`. + +- [ ] **Step 3: Link it from the footer** + +In `apps/website/src/components/shared/Footer.tsx`, add an `About` link to `/about` in the same group as the existing Contact link, matching the surrounding link markup exactly. + +- [ ] **Step 4: Verify** + +```bash +npx nx test website && npx nx build website +``` + +Expected: PASS and a successful build. Then serve and confirm `curl -s http://localhost:3000/about | grep -c 'application/ld+json'` returns `2`. + +- [ ] **Step 5: Commit** + +```bash +git add apps/website/src/app/about apps/website/src/lib/site-metadata.ts apps/website/src/components/shared/Footer.tsx +git commit -m "feat(website): about page with person entity" +``` + +--- + +### Task 12: Add diagrams to the blog + +Every blog post currently ships zero images. Google's guide asks for high-quality images and video where relevant, and a diagram is the highest-value image for an architecture tutorial. + +**Files:** +- Modify: `apps/website/content/blog/2026-08-13-angular-chat-app-tutorial-with-ag-ui.mdx` +- Modify: `apps/website/content/blog/2026-08-13-angular-chat-app-tutorial-with-langchain-langgraph.mdx` +- Modify: `apps/website/content/blog/2026-08-09-agentic-ui-in-angular-production-patterns.mdx` +- Create: `apps/website/public/blog/diagrams/*.svg` + +- [ ] **Step 1: Confirm the gap** + +```bash +grep -c "!\[" apps/website/content/blog/*.mdx +``` + +Expected: `0` for every file. + +- [ ] **Step 2: Check how MDX renders images** + +```bash +grep -n "img\|Image" apps/website/src/components/docs/MdxRenderer.tsx +``` + +If there is no `img` override, add one that renders a plain `` with `loading="lazy"`, `width`, `height`, and the `alt` from the MDX — dimensions prevent CLS, which is a page-experience signal. + +- [ ] **Step 3: Author one SVG diagram per post** + +For the AG-UI tutorial, create `apps/website/public/blog/diagrams/ag-ui-event-flow.svg` showing: AG-UI backend → SSE event stream (17 event types) → `@threadplane/ag-ui` adapter → Angular signals → `@threadplane/chat` components. Use the design tokens' surface and text colors so it reads in both themes. Repeat for the LangGraph tutorial (graph → SSE → adapter → signals) and the production-patterns post (client-tool continuation loop). + +- [ ] **Step 4: Reference each diagram with descriptive alt text** + +Insert immediately after the "What are we building?" heading of the AG-UI post: + +```mdx +![AG-UI backend streams 17 SSE event types into the @threadplane/ag-ui adapter, which converts them into Angular signals consumed by @threadplane/chat components.](/blog/diagrams/ag-ui-event-flow.svg) +``` + +The alt text is a full sentence describing the mechanism, not a label — it is extractable content, not decoration. + +- [ ] **Step 5: Verify** + +```bash +npx nx serve website +``` + +In a second shell: + +```bash +curl -s http://localhost:3000/blog/angular-chat-app-tutorial-with-ag-ui | grep -c "` has an `alt` attribute. + +- [ ] **Step 6: Commit** + +```bash +git add apps/website/content/blog apps/website/public/blog/diagrams apps/website/src/components/docs/MdxRenderer.tsx +git commit -m "docs(blog): architecture diagrams for the tutorial posts" +``` + +--- + +### Task 13: Question-form headings across older posts and key docs + +The two newest blog posts already use question-form H2s. The four older posts use noun phrases. Question headings match how people phrase things in AI Mode, and give retrieval a clean question/answer pair — without any "chunking" or keyword stuffing, both of which Google explicitly calls useless. + +**Files:** +- Modify: `apps/website/content/blog/2026-05-17-build-a-streaming-chat-ui-in-angular-with-langgraph.mdx` +- Modify: `apps/website/content/blog/2026-05-21-build-fullstack-agentic-angular-apps-using-ag-ui.mdx` +- Modify: `apps/website/content/blog/2026-05-28-human-in-the-loop-langgraph-agents-in-angular.mdx` +- Modify: `apps/website/content/blog/2026-06-04-human-in-the-loop-ag-ui-agents-in-angular.mdx` + +- [ ] **Step 1: List the current headings** + +```bash +grep -n "^## " apps/website/content/blog/2026-05-17-*.mdx apps/website/content/blog/2026-05-21-*.mdx apps/website/content/blog/2026-05-28-*.mdx apps/website/content/blog/2026-06-04-*.mdx +``` + +- [ ] **Step 2: Rewrite only the headings that are genuinely answering a question** + +Example, in the streaming post: `## Why streaming matters` → `## Why does streaming matter for agent UIs?`; `## The architecture in three boxes` → `## What does the architecture look like?`. Leave `## Goals` alone — it is not a question and forcing one would be exactly the "special rewrites for AI" Google warns against. Change no body copy. + +- [ ] **Step 3: Verify no anchors broke** + +```bash +grep -rn "](#" apps/website/content/blog | grep -v node_modules +``` + +Any in-page anchor whose target heading you renamed must be updated to the new slug. Expected after fixing: every listed anchor resolves to a heading present in the same file. + +- [ ] **Step 4: Commit** + +```bash +git add apps/website/content/blog +git commit -m "docs(blog): question-form section headings in the 2026-05/06 posts" +``` + +--- + +### Task 14: Freeze the programmatic `solutions/*` surface + +Three `solutions/*` pages generated from a 9.4 KB data file is fine. Thirty would be scaled content abuse, which Google's guide names as a policy violation. Write the rule down before someone scales it. + +**Files:** +- Modify: `apps/website/src/lib/solutions-data.ts` + +- [ ] **Step 1: Add the constraint as a file-header comment** + +At the top of `apps/website/src/lib/solutions-data.ts`, below the SPDX line: + +```ts +/** + * Solutions pages are hand-written, not templated at scale. + * + * Google's scaled-content-abuse policy targets programmatically generated + * page families that vary only by keyword. Every entry here must carry + * genuinely distinct, first-hand content — a real customer problem, a real + * architecture, real code. If a new entry would be a find-and-replace of an + * existing one, do not add it; write a blog post or a docs guide instead. + * + * See https://developers.google.com/search/docs/fundamentals/ai-optimization-guide + */ +``` + +- [ ] **Step 2: Verify each existing page is genuinely distinct** + +```bash +npx tsx -e " +import('./apps/website/src/lib/solutions-data.ts').then((m) => { + for (const s of m.solutions ?? []) console.log(s.slug, JSON.stringify(s).length); +}); +" +``` + +Read the three pages. If any two share more than boilerplate structure, rewrite the weaker one with genuinely different content or remove it from the sitemap. Record the decision in the PR description. + +- [ ] **Step 3: Commit** + +```bash +git add apps/website/src/lib/solutions-data.ts +git commit -m "docs(website): record the no-scaled-content rule for solutions pages" +``` + +--- + +## Phase 4 — Measurement + +### Task 15: AI crawler and AI referral tracking + +AI crawlers do not execute JavaScript, so the client PostHog snippet never sees them. Edge middleware does. This gives a real answer to "are AI engines reading us, and are they sending anyone back" — the one measurement Search Console will not provide. + +**Files:** +- Create: `apps/website/src/lib/analytics/ai-traffic.ts` +- Create: `apps/website/src/lib/analytics/ai-traffic.spec.ts` +- Modify: `apps/website/src/lib/analytics/events.ts` +- Create: `apps/website/src/middleware.ts` + +- [ ] **Step 1: Write the failing test** + +Create `apps/website/src/lib/analytics/ai-traffic.spec.ts`: + +```ts +import { describe, expect, it } from 'vitest'; +import { classifyAiCrawler, classifyAiReferrer } from './ai-traffic'; + +describe('classifyAiCrawler', () => { + it('identifies the major AI crawlers', () => { + expect(classifyAiCrawler('Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)')).toBe('gptbot'); + expect(classifyAiCrawler('Mozilla/5.0 (compatible; ClaudeBot/1.0)')).toBe('claudebot'); + expect(classifyAiCrawler('Mozilla/5.0 (compatible; PerplexityBot/1.0)')).toBe('perplexitybot'); + expect(classifyAiCrawler('Mozilla/5.0 (compatible; Google-Extended)')).toBe('google-extended'); + expect(classifyAiCrawler('Mozilla/5.0 (compatible; Googlebot/2.1)')).toBe(null); + expect(classifyAiCrawler('')).toBe(null); + }); +}); + +describe('classifyAiReferrer', () => { + it('identifies referrals from AI answer engines', () => { + expect(classifyAiReferrer('https://chatgpt.com/c/abc')).toBe('chatgpt'); + expect(classifyAiReferrer('https://www.perplexity.ai/search?q=x')).toBe('perplexity'); + expect(classifyAiReferrer('https://claude.ai/chat/1')).toBe('claude'); + expect(classifyAiReferrer('https://gemini.google.com/app')).toBe('gemini'); + expect(classifyAiReferrer('https://www.google.com/search?q=x')).toBe(null); + expect(classifyAiReferrer('')).toBe(null); + }); +}); +``` + +- [ ] **Step 2: Run it and confirm it fails** + +```bash +npx nx test website +``` + +Expected: FAIL — `Failed to resolve import "./ai-traffic"`. + +- [ ] **Step 3: Implement the classifiers** + +Create `apps/website/src/lib/analytics/ai-traffic.ts`: + +```ts +// SPDX-License-Identifier: MIT + +const CRAWLERS: [RegExp, string][] = [ + [/GPTBot/i, 'gptbot'], + [/OAI-SearchBot/i, 'oai-searchbot'], + [/ChatGPT-User/i, 'chatgpt-user'], + [/ClaudeBot/i, 'claudebot'], + [/Claude-Web/i, 'claude-web'], + [/anthropic-ai/i, 'anthropic-ai'], + [/PerplexityBot/i, 'perplexitybot'], + [/Perplexity-User/i, 'perplexity-user'], + [/Google-Extended/i, 'google-extended'], + [/Google-CloudVertexBot/i, 'google-vertex'], + [/Applebot-Extended/i, 'applebot-extended'], + [/Bytespider/i, 'bytespider'], + [/Meta-ExternalAgent/i, 'meta-external-agent'], + [/CCBot/i, 'ccbot'], +]; + +const REFERRERS: [RegExp, string][] = [ + [/(^|\.)chatgpt\.com$/i, 'chatgpt'], + [/(^|\.)chat\.openai\.com$/i, 'chatgpt'], + [/(^|\.)perplexity\.ai$/i, 'perplexity'], + [/(^|\.)claude\.ai$/i, 'claude'], + [/(^|\.)gemini\.google\.com$/i, 'gemini'], + [/(^|\.)copilot\.microsoft\.com$/i, 'copilot'], + [/(^|\.)you\.com$/i, 'you'], +]; + +/** Returns a stable crawler slug, or null for humans and non-AI bots. */ +export function classifyAiCrawler(userAgent: string): string | null { + if (!userAgent) return null; + for (const [pattern, slug] of CRAWLERS) { + if (pattern.test(userAgent)) return slug; + } + return null; +} + +/** Returns a stable AI-engine slug for a referrer URL, or null. */ +export function classifyAiReferrer(referrer: string): string | null { + if (!referrer) return null; + let host: string; + try { + host = new URL(referrer).hostname; + } catch { + return null; + } + for (const [pattern, slug] of REFERRERS) { + if (pattern.test(host)) return slug; + } + return null; +} +``` + +- [ ] **Step 4: Run the tests and confirm they pass** + +```bash +npx nx test website +``` + +Expected: PASS. + +- [ ] **Step 5: Register the events** + +Open `apps/website/src/lib/analytics/events.ts` and add two entries to the `analyticsEvents` registry, following the exact shape of the entries already there: + +```ts + aiCrawlerVisit: 'ai_crawler_visit', + aiReferralVisit: 'ai_referral_visit', +``` + +Use whatever key/value convention the existing entries use — read the file first and match it rather than assuming. + +- [ ] **Step 6: Write the middleware** + +Create `apps/website/src/middleware.ts`: + +```ts +// SPDX-License-Identifier: MIT +import { NextResponse, type NextRequest } from 'next/server'; +import { classifyAiCrawler, classifyAiReferrer } from './lib/analytics/ai-traffic'; + +export const config = { + // HTML routes only — skip assets, API routes, and Next internals. + matcher: ['/((?!api|_next/static|_next/image|ingest|favicon.ico|.*\\.(?:png|jpg|jpeg|svg|webp|ico|txt|xml|pdf)$).*)'], +}; + +const POSTHOG_HOST = 'https://us.i.posthog.com'; + +async function capture(event: string, properties: Record): Promise { + const token = process.env['NEXT_PUBLIC_POSTHOG_TOKEN']; + if (!token) return; + try { + await fetch(`${POSTHOG_HOST}/capture/`, { + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ + api_key: token, + event, + distinct_id: `ai:${properties['source']}`, + properties: { ...properties, $process_person_profile: false }, + }), + }); + } catch { + // Analytics must never break a page render. + } +} + +export function middleware(request: NextRequest): NextResponse { + const pathname = request.nextUrl.pathname; + const crawler = classifyAiCrawler(request.headers.get('user-agent') ?? ''); + const referrer = crawler ? null : classifyAiReferrer(request.headers.get('referer') ?? ''); + + if (crawler) { + void capture('ai_crawler_visit', { source: crawler, path: pathname }); + } else if (referrer) { + void capture('ai_referral_visit', { source: referrer, path: pathname }); + } + + return NextResponse.next(); +} +``` + +- [ ] **Step 7: Verify the middleware fires** + +```bash +npx nx serve website +``` + +In a second shell: + +```bash +curl -s -o /dev/null -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" http://localhost:3000/docs/chat/getting-started/installation +curl -s -o /dev/null -e "https://chatgpt.com/c/abc" http://localhost:3000/blog/angular-chat-app-tutorial-with-ag-ui +curl -s -o /dev/null http://localhost:3000/ +``` + +Expected: in PostHog's live-events view, exactly one `ai_crawler_visit` (source `gptbot`) and one `ai_referral_visit` (source `chatgpt`), and nothing for the third request. If `NEXT_PUBLIC_POSTHOG_TOKEN` is unset locally, assert instead that all three requests return 200 and no error is logged. + +- [ ] **Step 8: Confirm the production build still passes its bundle budget** + +```bash +npx nx build website --configuration=production +``` + +Expected: success. (This project has bitten prod deploys before with prod-only budgets that dev builds never check.) + +- [ ] **Step 9: Commit** + +```bash +git add apps/website/src/lib/analytics apps/website/src/middleware.ts +git commit -m "feat(website): track ai crawler and ai referral traffic" +``` + +--- + +### Task 16: Measurement runbook + +**Files:** +- Create: `docs/gtm/ai-search-measurement.md` + +- [ ] **Step 1: Write the runbook** + +Create `docs/gtm/ai-search-measurement.md`: + +```markdown +# Measuring AI search visibility for threadplane.ai + +## What each source can and cannot answer + +| Question | Source | Automated? | +|---|---|---| +| Impressions/clicks in AI Overviews and AI Mode | Search Console → Performance → **Generative AI** report | **No — UI only.** Not in `searchanalytics.query` (`type` accepts only web/image/video/news/discover/googleNews), not in `searchAppearance`, not in the BigQuery export, as of 2026-08. | +| Query and page performance in classic web search | `npm run gsc:pull && npm run gsc:report` | Yes | +| Index coverage per URL | URL Inspection API, via the same pull | Yes (quota 2000/day) | +| Discover impressions | `type: 'discover'` in the pull | Yes | +| Are AI crawlers fetching us | PostHog `ai_crawler_visit` | Yes | +| Are AI engines sending referrals | PostHog `ai_referral_visit` | Yes | +| Do we appear in a given assistant's answer | Manual spot-check | No | + +## Monthly routine + +1. `npm run gsc:pull && npm run gsc:report`, then read `apps/website/.gsc/report.md`. + Act on, in order: **not indexed** → **canonical mismatches** → **striking distance** → + **weak CTR**. +2. In Search Console, open Performance → Generative AI, set the window to the last + 3 months, and export the CSV. Save it as + `apps/website/.gsc/generative-ai-YYYY-MM.csv` (gitignored). Compare total + impressions and top pages against the previous month by hand. Re-check whether + the API has caught up — Google shipped Discover and News to the UI months before + the API, and this is expected to follow. +3. In PostHog, chart `ai_crawler_visit` by `source` and `path` over 90 days. A crawler + that stops fetching a section is an early warning. Chart `ai_referral_visit` by + `source` for the conversion side. +4. Spot-check five assistant prompts a real buyer would type — e.g. "Angular chat UI + for LangGraph", "how do I stream a LangGraph agent into Angular", "AG-UI Angular + client" — in ChatGPT, Claude, Perplexity, and Google AI Mode. Record whether + threadplane.ai is cited and what it is cited *for*. This is qualitative and slow, + and there is no honest way to automate it; do five, not fifty. + +## Tactics that do not work — do not add them + +Straight from Google's AI optimization guide: `llms.txt` and AI-specific text files do +nothing for Google Search; content "chunking" is unnecessary; AI-specific keyword +rewrites are unnecessary; pursuing inauthentic mentions is ineffective. Ignore any +third-party tool claiming access to internal Google AI metrics — those numbers are +modeled, not measured. + +We keep `/llms.txt` and `/llms-full.txt` because some non-Google assistants do read +them, not because they help Google. +``` + +- [ ] **Step 2: Verify the referenced npm scripts exist** + +```bash +grep -n "gsc:pull\|gsc:report" package.json +``` + +Expected: both present (added in Task 4). + +- [ ] **Step 3: Commit** + +```bash +git add docs/gtm/ai-search-measurement.md +git commit -m "docs(gtm): ai search measurement runbook" +``` + +--- + +## Final verification + +- [ ] **Step 1: Full check** + +```bash +npx nx test website && npx nx lint website && npx nx build website --configuration=production +``` + +Expected: all three pass. Lint **warnings** are tolerated by CI; lint **errors** are not. To count errors, strip ANSI first — `grep -cE ' error '` on raw output silently returns 0: + +```bash +npx nx lint website 2>&1 | sed -r "s/\x1B\[[0-9;]*[mK]//g" | grep -cE ' error ' +``` + +Expected: `0`. + +- [ ] **Step 2: Verify the live-equivalent output locally** + +```bash +npx nx serve website +``` + +In a second shell: + +```bash +for u in / /about /blog/angular-chat-app-tutorial-with-ag-ui /docs/chat/getting-started/installation; do + echo "=== $u" + curl -s "http://localhost:3000$u" -o /tmp/p.html + echo "ld+json blocks: $(grep -c 'application/ld+json' /tmp/p.html)" + grep -o '[^<]*' /tmp/p.html + grep -o '" +``` + +Expected: every page has at least one `ld+json` block, exactly one canonical, a title ending in `— Threadplane`, and the sitemap `lastmod` count equals the URL count (141 after `/about` lands). + +- [ ] **Step 3: Post-merge validation (record results in the PR)** + +1. Rich Results Test on the deployed `/blog/*` and `/docs/*` URLs → zero errors. +2. Search Console → Sitemaps → resubmit `sitemap.xml`. +3. Search Console → URL Inspection → Request indexing for `/about`. +4. Re-run `npm run gsc:pull && npm run gsc:report` 14 days later and diff the index-health section against the pre-change run. From 2ddc7884fef14afeca3f69e6aa7c29ea84d5cfe3 Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 20 Aug 2026 11:38:14 -0700 Subject: [PATCH 02/35] feat(website): search console api service-account auth Co-Authored-By: Claude Opus 5 --- apps/website/.gitignore | 1 + apps/website/scripts/gsc/README.md | 29 ++++++++++++++ apps/website/scripts/gsc/auth.ts | 61 ++++++++++++++++++++++++++++++ 3 files changed, 91 insertions(+) create mode 100644 apps/website/.gitignore create mode 100644 apps/website/scripts/gsc/README.md create mode 100644 apps/website/scripts/gsc/auth.ts diff --git a/apps/website/.gitignore b/apps/website/.gitignore new file mode 100644 index 000000000..bfcc80dd7 --- /dev/null +++ b/apps/website/.gitignore @@ -0,0 +1 @@ +.gsc/ diff --git a/apps/website/scripts/gsc/README.md b/apps/website/scripts/gsc/README.md new file mode 100644 index 000000000..e1763375b --- /dev/null +++ b/apps/website/scripts/gsc/README.md @@ -0,0 +1,29 @@ +# Search Console API harness + +## One-time setup + +1. In Google Cloud console, create (or reuse) a project and enable the + **Google Search Console API** (`searchconsole.googleapis.com`). +2. Create a service account. No project-level IAM roles are needed. +3. Create a JSON key for that service account and download it. +4. In Search Console, open the `threadplane.ai` **Domain property** → + Settings → Users and permissions → Add user → paste the service + account's `client_email` → permission **Full** (required: the URL + Inspection API rejects "Restricted" users). +5. Export the key for local use — do NOT commit it: + + export GSC_SERVICE_ACCOUNT_JSON="$(cat ~/secrets/threadplane-gsc.json)" + export GSC_SITE_URL="sc-domain:threadplane.ai" + +## Usage + + npx tsx apps/website/scripts/gsc/pull.ts # writes apps/website/.gsc/*.json + npx tsx apps/website/scripts/gsc/report.ts # writes apps/website/.gsc/report.md + +## What this CANNOT do + +The Search Console **Generative AI performance report** (AI Overviews / +AI Mode impressions and clicks) is UI-only as of 2026-08. It is not in +`searchanalytics.query`, not in `searchAppearance`, and not in the +BigQuery bulk export. See `docs/gtm/ai-search-measurement.md` for the +manual export procedure. diff --git a/apps/website/scripts/gsc/auth.ts b/apps/website/scripts/gsc/auth.ts new file mode 100644 index 000000000..f3a44c48f --- /dev/null +++ b/apps/website/scripts/gsc/auth.ts @@ -0,0 +1,61 @@ +// SPDX-License-Identifier: MIT +import { createSign } from 'node:crypto'; + +const TOKEN_URL = 'https://oauth2.googleapis.com/token'; +const SCOPE = 'https://www.googleapis.com/auth/webmasters.readonly'; + +interface ServiceAccountKey { + client_email: string; + private_key: string; +} + +function base64url(value: object): string { + return Buffer.from(JSON.stringify(value)).toString('base64url'); +} + +export function readServiceAccountKey(): ServiceAccountKey { + const raw = process.env['GSC_SERVICE_ACCOUNT_JSON']; + if (!raw) { + throw new Error( + 'GSC_SERVICE_ACCOUNT_JSON is not set. See apps/website/scripts/gsc/README.md.', + ); + } + const parsed = JSON.parse(raw) as Partial; + if (!parsed.client_email || !parsed.private_key) { + throw new Error('GSC_SERVICE_ACCOUNT_JSON is missing client_email or private_key.'); + } + return { client_email: parsed.client_email, private_key: parsed.private_key }; +} + +export async function getAccessToken(nowSeconds = Math.floor(Date.now() / 1000)): Promise { + const key = readServiceAccountKey(); + const signingInput = [ + base64url({ alg: 'RS256', typ: 'JWT' }), + base64url({ + iss: key.client_email, + scope: SCOPE, + aud: TOKEN_URL, + iat: nowSeconds, + exp: nowSeconds + 3600, + }), + ].join('.'); + const signature = createSign('RSA-SHA256') + .update(signingInput) + .sign(key.private_key, 'base64url'); + + const response = await fetch(TOKEN_URL, { + method: 'POST', + headers: { 'content-type': 'application/x-www-form-urlencoded' }, + body: new URLSearchParams({ + grant_type: 'urn:ietf:params:oauth:grant-type:jwt-bearer', + assertion: `${signingInput}.${signature}`, + }), + }); + + if (!response.ok) { + throw new Error(`Token exchange failed: ${response.status} ${await response.text()}`); + } + const json = (await response.json()) as { access_token?: string }; + if (!json.access_token) throw new Error('Token exchange returned no access_token.'); + return json.access_token; +} From deaa509d0650c0d4d4df251218a0291506c072fd Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 20 Aug 2026 11:42:36 -0700 Subject: [PATCH 03/35] feat(website): typed search console api wrappers Co-Authored-By: Claude Opus 5 --- apps/website/scripts/gsc/api.ts | 114 ++++++++++++++++++++++++++++++++ 1 file changed, 114 insertions(+) create mode 100644 apps/website/scripts/gsc/api.ts diff --git a/apps/website/scripts/gsc/api.ts b/apps/website/scripts/gsc/api.ts new file mode 100644 index 000000000..a15089c7e --- /dev/null +++ b/apps/website/scripts/gsc/api.ts @@ -0,0 +1,114 @@ +// SPDX-License-Identifier: MIT +import { getAccessToken } from './auth'; + +const BASE = 'https://www.googleapis.com/webmasters/v3'; +const INSPECT_URL = 'https://searchconsole.googleapis.com/v1/urlInspection/index:inspect'; + +export type Dimension = 'query' | 'page' | 'country' | 'device' | 'date' | 'searchAppearance'; + +export interface SearchAnalyticsRow { + keys: string[]; + clicks: number; + impressions: number; + ctr: number; + position: number; +} + +export function getSiteUrl(): string { + return process.env['GSC_SITE_URL'] ?? 'sc-domain:threadplane.ai'; +} + +async function authedFetch(url: string, init: RequestInit & { token: string }): Promise { + const { token, ...rest } = init; + const response = await fetch(url, { + ...rest, + headers: { ...(rest.headers ?? {}), authorization: `Bearer ${token}`, 'content-type': 'application/json' }, + }); + if (!response.ok) { + throw new Error(`${url} → ${response.status} ${await response.text()}`); + } + return response.json(); +} + +/** + * Search Analytics. NOTE: `type` accepts only web|image|video|news|discover| + * googleNews. There is no AI Overviews / AI Mode type as of 2026-08. + */ +export async function querySearchAnalytics(options: { + startDate: string; + endDate: string; + dimensions: Dimension[]; + rowLimit?: number; + startRow?: number; + type?: 'web' | 'image' | 'video' | 'news' | 'discover' | 'googleNews'; +}): Promise { + const token = await getAccessToken(); + const site = encodeURIComponent(getSiteUrl()); + const rows: SearchAnalyticsRow[] = []; + let startRow = options.startRow ?? 0; + const rowLimit = options.rowLimit ?? 25000; + + for (;;) { + const page = (await authedFetch(`${BASE}/sites/${site}/searchAnalytics/query`, { + token, + method: 'POST', + body: JSON.stringify({ + startDate: options.startDate, + endDate: options.endDate, + dimensions: options.dimensions, + type: options.type ?? 'web', + rowLimit, + startRow, + dataState: 'all', + }), + })) as { rows?: SearchAnalyticsRow[] }; + const batch = page.rows ?? []; + rows.push(...batch); + if (batch.length < rowLimit) break; + startRow += rowLimit; + } + return rows; +} + +export async function listSitemaps(): Promise { + const token = await getAccessToken(); + return authedFetch(`${BASE}/sites/${encodeURIComponent(getSiteUrl())}/sitemaps`, { + token, + method: 'GET', + }); +} + +export interface InspectionResult { + url: string; + verdict: string; + coverageState: string; + lastCrawlTime: string | null; + robotsTxtState: string; + indexingState: string; + googleCanonical: string | null; + userCanonical: string | null; +} + +export async function inspectUrl(inspectionUrl: string): Promise { + const token = await getAccessToken(); + const raw = (await authedFetch(INSPECT_URL, { + token, + method: 'POST', + body: JSON.stringify({ inspectionUrl, siteUrl: getSiteUrl(), languageCode: 'en-US' }), + })) as { + inspectionResult?: { + indexStatusResult?: Record; + }; + }; + const status = raw.inspectionResult?.indexStatusResult ?? {}; + return { + url: inspectionUrl, + verdict: status['verdict'] ?? 'UNKNOWN', + coverageState: status['coverageState'] ?? 'UNKNOWN', + lastCrawlTime: status['lastCrawlTime'] ?? null, + robotsTxtState: status['robotsTxtState'] ?? 'UNKNOWN', + indexingState: status['indexingState'] ?? 'UNKNOWN', + googleCanonical: status['googleCanonical'] ?? null, + userCanonical: status['userCanonical'] ?? null, + }; +} From 93b66174a26499fad64862749d8eb39a331e2fb3 Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 20 Aug 2026 11:46:16 -0700 Subject: [PATCH 04/35] fix(website): cache search console access tokens Co-Authored-By: Claude Opus 5 --- apps/website/scripts/gsc/auth.ts | 46 +++++++++++++++++++++++++++++--- 1 file changed, 43 insertions(+), 3 deletions(-) diff --git a/apps/website/scripts/gsc/auth.ts b/apps/website/scripts/gsc/auth.ts index f3a44c48f..d2b880750 100644 --- a/apps/website/scripts/gsc/auth.ts +++ b/apps/website/scripts/gsc/auth.ts @@ -4,11 +4,20 @@ import { createSign } from 'node:crypto'; const TOKEN_URL = 'https://oauth2.googleapis.com/token'; const SCOPE = 'https://www.googleapis.com/auth/webmasters.readonly'; +/** Refresh this many seconds before the token's actual expiry, not exactly at it. */ +const REFRESH_SKEW_SECONDS = 60; + interface ServiceAccountKey { client_email: string; private_key: string; } +interface CachedToken { + accessToken: string; + /** Epoch seconds at which the token stops being usable (server-reported expiry). */ + expiresAtSeconds: number; +} + function base64url(value: object): string { return Buffer.from(JSON.stringify(value)).toString('base64url'); } @@ -27,7 +36,19 @@ export function readServiceAccountKey(): ServiceAccountKey { return { client_email: parsed.client_email, private_key: parsed.private_key }; } -export async function getAccessToken(nowSeconds = Math.floor(Date.now() / 1000)): Promise { +let cachedToken: CachedToken | null = null; +let inFlightExchange: Promise | null = null; + +/** + * Clears the in-process access-token cache. Exists so tests (and long-lived + * callers that suspect a revoked/expired token) can force a fresh exchange. + */ +export function resetAccessTokenCache(): void { + cachedToken = null; + inFlightExchange = null; +} + +async function exchangeAccessToken(nowSeconds: number): Promise { const key = readServiceAccountKey(); const signingInput = [ base64url({ alg: 'RS256', typ: 'JWT' }), @@ -55,7 +76,26 @@ export async function getAccessToken(nowSeconds = Math.floor(Date.now() / 1000)) if (!response.ok) { throw new Error(`Token exchange failed: ${response.status} ${await response.text()}`); } - const json = (await response.json()) as { access_token?: string }; + const json = (await response.json()) as { access_token?: string; expires_in?: number }; if (!json.access_token) throw new Error('Token exchange returned no access_token.'); - return json.access_token; + return { + accessToken: json.access_token, + expiresAtSeconds: nowSeconds + (json.expires_in ?? 3600), + }; +} + +export async function getAccessToken(nowSeconds = Math.floor(Date.now() / 1000)): Promise { + if (cachedToken && cachedToken.expiresAtSeconds - REFRESH_SKEW_SECONDS > nowSeconds) { + return cachedToken.accessToken; + } + + if (!inFlightExchange) { + inFlightExchange = exchangeAccessToken(nowSeconds).finally(() => { + inFlightExchange = null; + }); + } + + const token = await inFlightExchange; + cachedToken = token; + return token.accessToken; } From 3ddf41b2c82110b9628ef8de4c4291a265d2b3af Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 20 Aug 2026 11:48:19 -0700 Subject: [PATCH 05/35] feat(website): search console snapshot puller Co-Authored-By: Claude Opus 5 --- apps/website/scripts/gsc/pull.ts | 57 ++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+) create mode 100644 apps/website/scripts/gsc/pull.ts diff --git a/apps/website/scripts/gsc/pull.ts b/apps/website/scripts/gsc/pull.ts new file mode 100644 index 000000000..46e1ccb41 --- /dev/null +++ b/apps/website/scripts/gsc/pull.ts @@ -0,0 +1,57 @@ +// SPDX-License-Identifier: MIT +import fs from 'node:fs'; +import path from 'node:path'; +import { inspectUrl, listSitemaps, querySearchAnalytics, type InspectionResult } from './api'; + +const OUT_DIR = path.join(process.cwd(), 'apps', 'website', '.gsc'); + +function isoDaysAgo(days: number): string { + const date = new Date(Date.now() - days * 86_400_000); + return date.toISOString().slice(0, 10); +} + +function write(name: string, value: unknown): void { + fs.mkdirSync(OUT_DIR, { recursive: true }); + fs.writeFileSync(path.join(OUT_DIR, name), JSON.stringify(value, null, 2)); + console.log(`wrote .gsc/${name}`); +} + +async function sitemapUrls(): Promise { + const response = await fetch('https://threadplane.ai/sitemap.xml'); + const xml = await response.text(); + return [...xml.matchAll(/([^<]+)<\/loc>/g)].map((m) => m[1]); +} + +async function main(): Promise { + // Search Analytics data lags ~2 days; end 3 days back for a stable window. + const endDate = isoDaysAgo(3); + const startDate = isoDaysAgo(93); + + write('meta.json', { startDate, endDate, pulledAt: new Date().toISOString() }); + write('queries.json', await querySearchAnalytics({ startDate, endDate, dimensions: ['query'] })); + write('pages.json', await querySearchAnalytics({ startDate, endDate, dimensions: ['page'] })); + write( + 'query-page.json', + await querySearchAnalytics({ startDate, endDate, dimensions: ['query', 'page'] }), + ); + write('dates.json', await querySearchAnalytics({ startDate, endDate, dimensions: ['date'] })); + write( + 'discover.json', + await querySearchAnalytics({ startDate, endDate, dimensions: ['page'], type: 'discover' }), + ); + write('sitemaps.json', await listSitemaps()); + + // URL Inspection is quota-limited (2000/day, 600/min). Serialize with a small delay. + const urls = await sitemapUrls(); + const inspections: InspectionResult[] = []; + for (const url of urls) { + inspections.push(await inspectUrl(url)); + await new Promise((resolve) => setTimeout(resolve, 150)); + } + write('inspections.json', inspections); +} + +main().catch((error: unknown) => { + console.error(error); + process.exitCode = 1; +}); From 3d3f3d2274e0e68fa169dc42f4a60928a7d52156 Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 20 Aug 2026 11:52:45 -0700 Subject: [PATCH 06/35] fix(website): keep partial inspection results and fail loudly on a bad sitemap Co-Authored-By: Claude Opus 5 --- apps/website/scripts/gsc/pull.ts | 36 +++++++++++++++++++++++++++++--- 1 file changed, 33 insertions(+), 3 deletions(-) diff --git a/apps/website/scripts/gsc/pull.ts b/apps/website/scripts/gsc/pull.ts index 46e1ccb41..23971c178 100644 --- a/apps/website/scripts/gsc/pull.ts +++ b/apps/website/scripts/gsc/pull.ts @@ -18,8 +18,20 @@ function write(name: string, value: unknown): void { async function sitemapUrls(): Promise { const response = await fetch('https://threadplane.ai/sitemap.xml'); + if (!response.ok) { + throw new Error(`sitemap fetch failed: ${response.status} ${response.statusText}`); + } const xml = await response.text(); - return [...xml.matchAll(/([^<]+)<\/loc>/g)].map((m) => m[1]); + const urls = [...xml.matchAll(/([^<]+)<\/loc>/g)].map((m) => m[1]); + if (urls.length === 0) { + throw new Error('sitemap.xml matched zero entries; treating as a broken fetch, not empty inventory.'); + } + return urls; +} + +interface InspectionFailure { + url: string; + error: string; } async function main(): Promise { @@ -42,13 +54,31 @@ async function main(): Promise { write('sitemaps.json', await listSitemaps()); // URL Inspection is quota-limited (2000/day, 600/min). Serialize with a small delay. + // Failures are collected separately rather than dropped or faked, so a partial + // sweep still yields a usable inspections.json plus a visible error trail. const urls = await sitemapUrls(); const inspections: InspectionResult[] = []; - for (const url of urls) { - inspections.push(await inspectUrl(url)); + const failures: InspectionFailure[] = []; + for (const [index, url] of urls.entries()) { + try { + inspections.push(await inspectUrl(url)); + } catch (error) { + failures.push({ url, error: error instanceof Error ? error.message : String(error) }); + } + const done = index + 1; + if (done % 25 === 0 || done === urls.length) { + console.log(`inspected ${done}/${urls.length} urls`); + } await new Promise((resolve) => setTimeout(resolve, 150)); } write('inspections.json', inspections); + if (failures.length > 0) { + write('inspection-errors.json', failures); + } + console.log( + `inspected ${inspections.length}/${urls.length} urls, ${failures.length} failed` + + (failures.length > 0 ? ' (see .gsc/inspection-errors.json)' : ''), + ); } main().catch((error: unknown) => { From 5852de3d9efebf77ba653cbbf99b43fc9344670f Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 20 Aug 2026 12:01:33 -0700 Subject: [PATCH 07/35] feat(website): search console analysis report Pure analysis helpers (striking distance, zero-impression pages, unindexed, canonical mismatches, weak CTR) plus a markdown report generator over the .gsc snapshots. The report reads inspection-errors.json when present so a partial inspection sweep is stated as partial: failed URLs stay in the sitemap inventory, the failure count is reported, and the index-health counts are labelled lower bounds rather than implying a clean bill of health. Co-Authored-By: Claude Opus 5 --- apps/website/scripts/gsc/analysis.spec.ts | 88 +++++++++++++ apps/website/scripts/gsc/analysis.ts | 76 +++++++++++ apps/website/scripts/gsc/report.ts | 150 ++++++++++++++++++++++ apps/website/vite.config.mts | 2 +- package.json | 2 + 5 files changed, 317 insertions(+), 1 deletion(-) create mode 100644 apps/website/scripts/gsc/analysis.spec.ts create mode 100644 apps/website/scripts/gsc/analysis.ts create mode 100644 apps/website/scripts/gsc/report.ts diff --git a/apps/website/scripts/gsc/analysis.spec.ts b/apps/website/scripts/gsc/analysis.spec.ts new file mode 100644 index 000000000..4e048f457 --- /dev/null +++ b/apps/website/scripts/gsc/analysis.spec.ts @@ -0,0 +1,88 @@ +import { describe, expect, it } from 'vitest'; +import { + capList, + describeInspectionCoverage, + findStrikingDistance, + findUnindexed, + findZeroImpressionPages, +} from './analysis'; + +const rows = [ + { keys: ['angular langgraph chat'], clicks: 0, impressions: 400, ctr: 0, position: 11.2 }, + { keys: ['threadplane'], clicks: 90, impressions: 100, ctr: 0.9, position: 1.1 }, + { keys: ['obscure long tail'], clicks: 0, impressions: 3, ctr: 0, position: 42 }, +]; + +describe('findStrikingDistance', () => { + it('returns rows ranking 5-20 with meaningful impressions, best opportunity first', () => { + const result = findStrikingDistance(rows, { minImpressions: 50 }); + expect(result.map((r) => r.keys[0])).toEqual(['angular langgraph chat']); + }); +}); + +describe('findZeroImpressionPages', () => { + it('lists sitemap URLs that earned no impressions in the window', () => { + const result = findZeroImpressionPages( + ['https://threadplane.ai/a', 'https://threadplane.ai/b'], + [{ keys: ['https://threadplane.ai/a'], clicks: 1, impressions: 10, ctr: 0.1, position: 5 }], + ); + expect(result).toEqual(['https://threadplane.ai/b']); + }); +}); + +describe('findUnindexed', () => { + it('flags inspections whose verdict is not PASS', () => { + const result = findUnindexed([ + { + url: 'https://threadplane.ai/a', + verdict: 'PASS', + coverageState: 'Submitted and indexed', + lastCrawlTime: null, + robotsTxtState: 'ALLOWED', + indexingState: 'INDEXING_ALLOWED', + googleCanonical: null, + userCanonical: null, + }, + { + url: 'https://threadplane.ai/b', + verdict: 'NEUTRAL', + coverageState: 'Discovered - currently not indexed', + lastCrawlTime: null, + robotsTxtState: 'ALLOWED', + indexingState: 'INDEXING_ALLOWED', + googleCanonical: null, + userCanonical: null, + }, + ]); + expect(result.map((r) => r.url)).toEqual(['https://threadplane.ai/b']); + }); +}); + +describe('describeInspectionCoverage', () => { + it('reports a complete sweep when nothing failed', () => { + expect(describeInspectionCoverage({ inspected: 42, failed: 0 })).toBe( + 'Coverage: complete — all 42 sitemap URLs inspected.', + ); + }); + + it('warns that counts are lower bounds when inspections failed', () => { + const text = describeInspectionCoverage({ inspected: 8, failed: 2 }); + expect(text).toContain('PARTIAL'); + expect(text).toContain('2 of 10'); + expect(text).toContain('lower bound'); + }); +}); + +describe('capList', () => { + it('passes a short list through with nothing withheld', () => { + expect(capList(['a', 'b'], 20)).toEqual({ shown: ['a', 'b'], remaining: 0 }); + }); + + it('trims an overflowing list and counts the remainder', () => { + const items = Array.from({ length: 23 }, (_, i) => `url-${i}`); + const result = capList(items, 20); + expect(result.shown).toHaveLength(20); + expect(result.shown[19]).toBe('url-19'); + expect(result.remaining).toBe(3); + }); +}); diff --git a/apps/website/scripts/gsc/analysis.ts b/apps/website/scripts/gsc/analysis.ts new file mode 100644 index 000000000..9c834bc14 --- /dev/null +++ b/apps/website/scripts/gsc/analysis.ts @@ -0,0 +1,76 @@ +// SPDX-License-Identifier: MIT +import type { InspectionResult, SearchAnalyticsRow } from './api'; + +/** Queries ranking just off page one — the cheapest ranking wins available. */ +export function findStrikingDistance( + rows: SearchAnalyticsRow[], + options: { minImpressions: number }, +): SearchAnalyticsRow[] { + return rows + .filter( + (row) => + row.position >= 5 && row.position <= 20 && row.impressions >= options.minImpressions, + ) + .sort((a, b) => b.impressions - a.impressions); +} + +/** Sitemap URLs Google never showed for anything in the window. */ +export function findZeroImpressionPages( + sitemapUrls: string[], + pageRows: SearchAnalyticsRow[], +): string[] { + const seen = new Set(pageRows.map((row) => row.keys[0].replace(/\/$/, ''))); + return sitemapUrls.filter((url) => !seen.has(url.replace(/\/$/, ''))); +} + +/** Inspections that are not cleanly indexed. */ +export function findUnindexed(inspections: InspectionResult[]): InspectionResult[] { + return inspections.filter((inspection) => inspection.verdict !== 'PASS'); +} + +/** Pages Google canonicalized somewhere other than where we asked — duplicate-content smell. */ +export function findCanonicalMismatches(inspections: InspectionResult[]): InspectionResult[] { + return inspections.filter( + (inspection) => + inspection.googleCanonical !== null && + inspection.userCanonical !== null && + inspection.googleCanonical !== inspection.userCanonical, + ); +} + +/** Queries with strong impressions but a CTR well below the position-typical rate. */ +export function findWeakCtr( + rows: SearchAnalyticsRow[], + options: { minImpressions: number; maxCtr: number }, +): SearchAnalyticsRow[] { + return rows + .filter( + (row) => + row.impressions >= options.minImpressions && + row.position <= 10 && + row.ctr < options.maxCtr, + ) + .sort((a, b) => b.impressions - a.impressions); +} + +/** + * How complete an inspection sweep was. `pull.ts` records every URL Inspection + * failure in `inspection-errors.json`, so a sweep can cover fewer URLs than the + * sitemap lists — in which case every index-health count is a lower bound and + * the report has to say so rather than imply a clean bill of health. + */ +export function describeInspectionCoverage(counts: { inspected: number; failed: number }): string { + const total = counts.inspected + counts.failed; + if (counts.failed === 0) { + return `Coverage: complete — all ${total} sitemap URLs inspected.`; + } + return ( + `Coverage: PARTIAL — ${counts.failed} of ${total} sitemap URLs could not be inspected. ` + + `Every count below is a lower bound: an uninspected page may also be unindexed or canonicalized elsewhere.` + ); +} + +/** Trim a list for display, reporting how much was withheld. */ +export function capList(items: string[], limit: number): { shown: string[]; remaining: number } { + return { shown: items.slice(0, limit), remaining: Math.max(0, items.length - limit) }; +} diff --git a/apps/website/scripts/gsc/report.ts b/apps/website/scripts/gsc/report.ts new file mode 100644 index 000000000..52167b2ea --- /dev/null +++ b/apps/website/scripts/gsc/report.ts @@ -0,0 +1,150 @@ +// SPDX-License-Identifier: MIT +import fs from 'node:fs'; +import path from 'node:path'; +import type { InspectionResult, SearchAnalyticsRow } from './api'; +import { + capList, + describeInspectionCoverage, + findCanonicalMismatches, + findStrikingDistance, + findUnindexed, + findWeakCtr, + findZeroImpressionPages, +} from './analysis'; + +const DIR = path.join(process.cwd(), 'apps', 'website', '.gsc'); + +/** A URL Inspection call that failed during the pull, as recorded by pull.ts. */ +interface InspectionFailure { + url: string; + error: string; +} + +function read(name: string): T { + return JSON.parse(fs.readFileSync(path.join(DIR, name), 'utf8')) as T; +} + +/** Like `read`, for a file the pull only writes on a partial sweep. Absent is null; nothing else is swallowed. */ +function readOptional(name: string): T | null { + try { + return read(name); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === 'ENOENT') { + return null; + } + throw error; + } +} + +function table(rows: SearchAnalyticsRow[], headers: string[], limit = 30): string { + const head = `| ${headers.join(' | ')} |\n| ${headers.map(() => '---').join(' | ')} |`; + const body = rows + .slice(0, limit) + .map( + (row) => + `| ${row.keys.join(' | ')} | ${row.clicks} | ${row.impressions} | ${(row.ctr * 100).toFixed(1)}% | ${row.position.toFixed(1)} |`, + ) + .join('\n'); + return `${head}\n${body}`; +} + +function main(): void { + const meta = read<{ startDate: string; endDate: string }>('meta.json'); + const queries = read('queries.json'); + const pages = read('pages.json'); + const inspections = read('inspections.json'); + const failures = readOptional('inspection-errors.json') ?? []; + // Sitemap inventory = URLs we inspected PLUS URLs we failed to inspect, so a + // failed URL still reaches the zero-impression analysis instead of vanishing. + const sitemapUrls = [...inspections.map((i) => i.url), ...failures.map((f) => f.url)]; + + const totals = queries.reduce( + (acc, row) => ({ + clicks: acc.clicks + row.clicks, + impressions: acc.impressions + row.impressions, + }), + { clicks: 0, impressions: 0 }, + ); + + const unindexed = findUnindexed(inspections); + const mismatches = findCanonicalMismatches(inspections); + const orphans = findZeroImpressionPages(sitemapUrls, pages); + const failed = capList( + failures.map((failure) => `${failure.url} — ${failure.error}`), + 20, + ); + + const report = [ + `# threadplane.ai — Search Console report`, + ``, + `Window: ${meta.startDate} → ${meta.endDate}. Total clicks ${totals.clicks}, impressions ${totals.impressions}.`, + ``, + `> Google's AI Overviews / AI Mode impressions are NOT included — that report is UI-only.`, + `> See docs/gtm/ai-search-measurement.md.`, + ``, + `## Index health`, + ``, + describeInspectionCoverage({ inspected: inspections.length, failed: failures.length }), + ``, + `- Sitemap URLs inspected: ${inspections.length}`, + `- Inspections that failed: ${failures.length}`, + `- Not cleanly indexed: ${unindexed.length}`, + `- Google canonical ≠ our canonical: ${mismatches.length}`, + `- Zero-impression pages in window: ${orphans.length}`, + ``, + ...(failures.length > 0 + ? [ + `### Failed inspections`, + ``, + failed.shown.map((line) => `- ${line}`).join('\n'), + ...(failed.remaining > 0 ? [``, `_…and ${failed.remaining} more._`] : []), + ``, + ] + : []), + `### Not indexed`, + ``, + unindexed.length ? unindexed.map((i) => `- ${i.url} — ${i.coverageState}`).join('\n') : '_none_', + ``, + `### Canonical mismatches`, + ``, + mismatches.length + ? mismatches.map((i) => `- ${i.url} → Google chose ${i.googleCanonical}`).join('\n') + : '_none_', + ``, + `### Zero-impression pages`, + ``, + orphans.length ? orphans.map((url) => `- ${url}`).join('\n') : '_none_', + ``, + `## Striking distance (position 5–20, ≥50 impressions)`, + ``, + table(findStrikingDistance(queries, { minImpressions: 50 }), [ + 'Query', + 'Clicks', + 'Impr', + 'CTR', + 'Pos', + ]), + ``, + `## Weak CTR on page one (≥100 impressions, CTR < 2%)`, + ``, + `Title/description rewrite candidates.`, + ``, + table(findWeakCtr(queries, { minImpressions: 100, maxCtr: 0.02 }), [ + 'Query', + 'Clicks', + 'Impr', + 'CTR', + 'Pos', + ]), + ``, + `## Top pages`, + ``, + table(pages.slice(0, 30), ['Page', 'Clicks', 'Impr', 'CTR', 'Pos']), + ``, + ].join('\n'); + + fs.writeFileSync(path.join(DIR, 'report.md'), report); + console.log('wrote .gsc/report.md'); +} + +main(); diff --git a/apps/website/vite.config.mts b/apps/website/vite.config.mts index 29ba3befa..2fd9a3a69 100644 --- a/apps/website/vite.config.mts +++ b/apps/website/vite.config.mts @@ -10,6 +10,6 @@ export default defineConfig({ test: { environment: 'jsdom', globals: true, - include: ['src/**/*.spec.ts', 'src/**/*.spec.tsx'], + include: ['src/**/*.spec.ts', 'src/**/*.spec.tsx', 'scripts/**/*.spec.ts'], }, }); diff --git a/package.json b/package.json index a1bb71ae0..f413e66b6 100644 --- a/package.json +++ b/package.json @@ -4,6 +4,8 @@ "license": "MIT", "scripts": { "postinstall": "node libs/licensing/scripts/generate-public-key.mjs", + "gsc:pull": "npx tsx apps/website/scripts/gsc/pull.ts", + "gsc:report": "npx tsx apps/website/scripts/gsc/report.ts", "generate-agent-context": "npx tsx --tsconfig apps/website/tsconfig.json apps/website/scripts/generate-agent-context.ts", "generate-api-docs": "npx tsx apps/website/scripts/generate-api-docs.ts", "generate-narrative-docs": "npx tsx apps/website/scripts/generate-narrative-docs.ts", From c7cd9078485c8d44f3997a4a24f201a84fd9704e Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 20 Aug 2026 12:13:24 -0700 Subject: [PATCH 08/35] test(website): cover analysis boundaries and harden the gsc report Tests: real ordering assertions for findStrikingDistance and findWeakCtr (the previous one-row fixture passed with .sort() deleted), table-driven filter boundaries including the inclusive/exclusive asymmetry between impressions and ctr, and first coverage for findCanonicalMismatches and findWeakCtr. The both-canonicals-required policy is now pinned by test and stated in the doc comment. Report: InspectionFailure moves to api.ts so the pull/report serialization contract is declared once; snapshot reads report a missing or corrupt file by name and point at the pull instead of throwing a raw ENOENT, with readOptional keeping its own existence check so genuine absence stays distinguishable; URL comparison normalizes protocol, host case, www, fragment and query string so a tagged URL is no longer reported as a page with zero impressions; all four bullet lists are capped, not just the failures. Co-Authored-By: Claude Opus 5 --- apps/website/scripts/gsc/README.md | 17 ++- apps/website/scripts/gsc/analysis.spec.ts | 142 +++++++++++++++++++++ apps/website/scripts/gsc/analysis.ts | 52 +++++++- apps/website/scripts/gsc/api.ts | 11 ++ apps/website/scripts/gsc/pull.ts | 13 +- apps/website/scripts/gsc/report.ts | 103 +++++++-------- apps/website/scripts/gsc/snapshots.spec.ts | 48 +++++++ apps/website/scripts/gsc/snapshots.ts | 41 ++++++ 8 files changed, 360 insertions(+), 67 deletions(-) create mode 100644 apps/website/scripts/gsc/snapshots.spec.ts create mode 100644 apps/website/scripts/gsc/snapshots.ts diff --git a/apps/website/scripts/gsc/README.md b/apps/website/scripts/gsc/README.md index e1763375b..b82898740 100644 --- a/apps/website/scripts/gsc/README.md +++ b/apps/website/scripts/gsc/README.md @@ -17,8 +17,21 @@ ## Usage - npx tsx apps/website/scripts/gsc/pull.ts # writes apps/website/.gsc/*.json - npx tsx apps/website/scripts/gsc/report.ts # writes apps/website/.gsc/report.md + npm run gsc:pull # writes apps/website/.gsc/*.json + npm run gsc:report # writes apps/website/.gsc/report.md + +`gsc:report` reads the snapshots `gsc:pull` wrote, so run the pull first. + +Both scripts resolve their `.gsc` directory relative to the current working +directory (`/apps/website/.gsc`), so the raw form must be invoked from the +repo root: + + npx tsx apps/website/scripts/gsc/pull.ts + npx tsx apps/website/scripts/gsc/report.ts + +If the URL Inspection sweep hits quota or transient errors, `pull.ts` also +writes `.gsc/inspection-errors.json` and the report labels its index-health +counts as lower bounds. ## What this CANNOT do diff --git a/apps/website/scripts/gsc/analysis.spec.ts b/apps/website/scripts/gsc/analysis.spec.ts index 4e048f457..91cbe7487 100644 --- a/apps/website/scripts/gsc/analysis.spec.ts +++ b/apps/website/scripts/gsc/analysis.spec.ts @@ -1,12 +1,42 @@ import { describe, expect, it } from 'vitest'; +import type { InspectionResult } from './api'; import { capList, describeInspectionCoverage, + findCanonicalMismatches, findStrikingDistance, findUnindexed, + findWeakCtr, findZeroImpressionPages, + normalizeUrl, } from './analysis'; +/** A row with sane defaults, so each test states only the field it is about. */ +function row(overrides: Partial<{ key: string; clicks: number; impressions: number; ctr: number; position: number }>) { + return { + keys: [overrides.key ?? 'q'], + clicks: overrides.clicks ?? 0, + impressions: overrides.impressions ?? 500, + ctr: overrides.ctr ?? 0, + position: overrides.position ?? 8, + }; +} + +/** An inspection with sane defaults, so each test states only the field it is about. */ +function inspection(overrides: Partial): InspectionResult { + return { + url: 'https://threadplane.ai/a', + verdict: 'PASS', + coverageState: 'Submitted and indexed', + lastCrawlTime: null, + robotsTxtState: 'ALLOWED', + indexingState: 'INDEXING_ALLOWED', + googleCanonical: null, + userCanonical: null, + ...overrides, + }; +} + const rows = [ { keys: ['angular langgraph chat'], clicks: 0, impressions: 400, ctr: 0, position: 11.2 }, { keys: ['threadplane'], clicks: 90, impressions: 100, ctr: 0.9, position: 1.1 }, @@ -18,6 +48,102 @@ describe('findStrikingDistance', () => { const result = findStrikingDistance(rows, { minImpressions: 50 }); expect(result.map((r) => r.keys[0])).toEqual(['angular langgraph chat']); }); + + it('orders surviving rows by descending impressions, not input order', () => { + const result = findStrikingDistance( + [ + row({ key: 'middle', impressions: 200 }), + row({ key: 'smallest', impressions: 60 }), + row({ key: 'biggest', impressions: 900 }), + ], + { minImpressions: 50 }, + ); + expect(result.map((r) => r.keys[0])).toEqual(['biggest', 'middle', 'smallest']); + }); + + it.each([ + ['position exactly 5 is included', { position: 5 }, true], + ['position exactly 20 is included', { position: 20 }, true], + ['position just above page one at 4.9 is excluded', { position: 4.9 }, false], + ['position just past 20 at 20.1 is excluded', { position: 20.1 }, false], + ['impressions exactly at the floor are included', { impressions: 50 }, true], + ['impressions one below the floor are excluded', { impressions: 49 }, false], + ])('%s', (_name, overrides, kept) => { + const result = findStrikingDistance([row(overrides)], { minImpressions: 50 }); + expect(result).toHaveLength(kept ? 1 : 0); + }); +}); + +describe('findWeakCtr', () => { + it('orders surviving rows by descending impressions', () => { + const result = findWeakCtr( + [ + row({ key: 'fewer', impressions: 150, ctr: 0.01, position: 3 }), + row({ key: 'more', impressions: 800, ctr: 0.01, position: 3 }), + ], + { minImpressions: 100, maxCtr: 0.02 }, + ); + expect(result.map((r) => r.keys[0])).toEqual(['more', 'fewer']); + }); + + it.each([ + ['position exactly 10 is included', { position: 10, ctr: 0.01 }, true], + ['position 10.1 is off page one and excluded', { position: 10.1, ctr: 0.01 }, false], + ['impressions exactly at the floor are included', { impressions: 100, ctr: 0.01 }, true], + ['impressions one below the floor are excluded', { impressions: 99, ctr: 0.01 }, false], + ['ctr exactly at the ceiling is EXCLUDED — the bound is strict', { ctr: 0.02 }, false], + ['ctr just under the ceiling is included', { ctr: 0.0199 }, true], + ])('%s', (_name, overrides, kept) => { + const result = findWeakCtr([row({ position: 3, impressions: 500, ...overrides })], { + minImpressions: 100, + maxCtr: 0.02, + }); + expect(result).toHaveLength(kept ? 1 : 0); + }); +}); + +describe('findCanonicalMismatches', () => { + it('flags a page Google canonicalized away from our declared canonical', () => { + const result = findCanonicalMismatches([ + inspection({ + url: 'https://threadplane.ai/pricing', + googleCanonical: 'https://threadplane.ai/plans', + userCanonical: 'https://threadplane.ai/pricing', + }), + ]); + expect(result.map((i) => i.url)).toEqual(['https://threadplane.ai/pricing']); + }); + + it.each([ + ['both canonicals absent', null, null], + ['Google reports one but we declared none — deliberately NOT a mismatch', '/g', null], + ['we declared one but Google reports none', null, '/u'], + ['both present and in agreement', '/same', '/same'], + ])('does not flag when %s', (_name, googleCanonical, userCanonical) => { + expect(findCanonicalMismatches([inspection({ googleCanonical, userCanonical })])).toEqual([]); + }); +}); + +describe('normalizeUrl', () => { + it.each([ + ['protocol differs', 'http://threadplane.ai/blog/foo', 'https://threadplane.ai/blog/foo'], + ['host case differs', 'https://ThreadPlane.AI/blog/foo', 'https://threadplane.ai/blog/foo'], + ['leading www differs', 'https://www.threadplane.ai/blog/foo', 'https://threadplane.ai/blog/foo'], + ['trailing slash differs', 'https://threadplane.ai/blog/foo/', 'https://threadplane.ai/blog/foo'], + ['a fragment is present', 'https://threadplane.ai/blog/foo#intro', 'https://threadplane.ai/blog/foo'], + ['a query string is present', 'https://threadplane.ai/blog/foo?ref=x', 'https://threadplane.ai/blog/foo'], + ])('treats two URLs as the same page when %s', (_name, a, b) => { + expect(normalizeUrl(a)).toBe(normalizeUrl(b)); + }); + + it('keeps genuinely different paths apart', () => { + expect(normalizeUrl('https://threadplane.ai/a')).not.toBe(normalizeUrl('https://threadplane.ai/b')); + }); + + it('falls back to a textual cleanup instead of throwing on unparseable input', () => { + expect(normalizeUrl(' not a url?ref=x ')).toBe('not a url'); + expect(normalizeUrl('')).toBe(''); + }); }); describe('findZeroImpressionPages', () => { @@ -28,6 +154,22 @@ describe('findZeroImpressionPages', () => { ); expect(result).toEqual(['https://threadplane.ai/b']); }); + + it('does not report a page as invisible just because Search Console tagged the URL', () => { + const result = findZeroImpressionPages( + ['https://threadplane.ai/blog/foo'], + [ + { + keys: ['https://www.threadplane.ai/blog/foo/?ref=newsletter#top'], + clicks: 4, + impressions: 90, + ctr: 0.044, + position: 6, + }, + ], + ); + expect(result).toEqual([]); + }); }); describe('findUnindexed', () => { diff --git a/apps/website/scripts/gsc/analysis.ts b/apps/website/scripts/gsc/analysis.ts index 9c834bc14..6c8f8b980 100644 --- a/apps/website/scripts/gsc/analysis.ts +++ b/apps/website/scripts/gsc/analysis.ts @@ -14,13 +14,41 @@ export function findStrikingDistance( .sort((a, b) => b.impressions - a.impressions); } +/** + * Reduce a URL to the identity we compare on: no protocol, no `www.`, lowercased + * host, no trailing slash, no fragment, and NO QUERY STRING. + * + * Dropping the query string is a judgement call. Search Console's `page` + * dimension reports campaign- and referral-tagged URLs (`/blog/foo?ref=x`) + * that never appear in sitemap `` entries, and treating those as separate + * pages would report a page with real impressions as invisible. The cost is + * that a site where the query string genuinely selects content (`?page=2`, + * `?id=`) would collapse distinct pages together; threadplane.ai has no such + * routes. Total function — unparseable input falls back to a textual cleanup + * rather than throwing, since a single odd row must not kill the report. + */ +export function normalizeUrl(raw: string): string { + const trimmed = raw.trim(); + try { + const url = new URL(trimmed); + return `${url.hostname.toLowerCase().replace(/^www\./, '')}${url.pathname.replace(/\/$/, '')}`; + } catch { + return trimmed + .toLowerCase() + .replace(/^[a-z][a-z0-9+.-]*:\/\//, '') + .replace(/^www\./, '') + .replace(/[?#].*$/, '') + .replace(/\/$/, ''); + } +} + /** Sitemap URLs Google never showed for anything in the window. */ export function findZeroImpressionPages( sitemapUrls: string[], pageRows: SearchAnalyticsRow[], ): string[] { - const seen = new Set(pageRows.map((row) => row.keys[0].replace(/\/$/, ''))); - return sitemapUrls.filter((url) => !seen.has(url.replace(/\/$/, ''))); + const seen = new Set(pageRows.map((row) => normalizeUrl(row.keys[0]))); + return sitemapUrls.filter((url) => !seen.has(normalizeUrl(url))); } /** Inspections that are not cleanly indexed. */ @@ -28,7 +56,15 @@ export function findUnindexed(inspections: InspectionResult[]): InspectionResult return inspections.filter((inspection) => inspection.verdict !== 'PASS'); } -/** Pages Google canonicalized somewhere other than where we asked — duplicate-content smell. */ +/** + * Pages Google canonicalized somewhere other than where we asked — duplicate-content smell. + * + * Policy: BOTH canonicals must be present. A page where Google reports a + * canonical but `userCanonical` is null (i.e. we emitted no ``) + * is deliberately NOT flagged here — it is a finding in its own right, but a + * different one, and folding it in would make "mismatch" mean two things. It is + * still visible in the raw `.gsc/inspections.json` snapshot. + */ export function findCanonicalMismatches(inspections: InspectionResult[]): InspectionResult[] { return inspections.filter( (inspection) => @@ -38,7 +74,15 @@ export function findCanonicalMismatches(inspections: InspectionResult[]): Inspec ); } -/** Queries with strong impressions but a CTR well below the position-typical rate. */ +/** + * Queries with strong impressions that sit on page one yet convert below one + * flat CTR threshold. + * + * Limitation: the threshold does not vary with position, so a 1.9% CTR at + * position 1 (alarming) is reported identically to 1.9% at position 10 + * (unremarkable). Read the Pos column before acting; ranking the output by + * position-relative expected CTR would need a baseline curve we do not have. + */ export function findWeakCtr( rows: SearchAnalyticsRow[], options: { minImpressions: number; maxCtr: number }, diff --git a/apps/website/scripts/gsc/api.ts b/apps/website/scripts/gsc/api.ts index a15089c7e..15555d73a 100644 --- a/apps/website/scripts/gsc/api.ts +++ b/apps/website/scripts/gsc/api.ts @@ -89,6 +89,17 @@ export interface InspectionResult { userCanonical: string | null; } +/** + * A URL Inspection call that failed during a pull. Serialization contract: + * `pull.ts` writes these to `.gsc/inspection-errors.json`, `report.ts` reads + * them back — so the shape lives here, next to InspectionResult, rather than + * being restated at each end. + */ +export interface InspectionFailure { + url: string; + error: string; +} + export async function inspectUrl(inspectionUrl: string): Promise { const token = await getAccessToken(); const raw = (await authedFetch(INSPECT_URL, { diff --git a/apps/website/scripts/gsc/pull.ts b/apps/website/scripts/gsc/pull.ts index 23971c178..2d787e6f2 100644 --- a/apps/website/scripts/gsc/pull.ts +++ b/apps/website/scripts/gsc/pull.ts @@ -1,7 +1,13 @@ // SPDX-License-Identifier: MIT import fs from 'node:fs'; import path from 'node:path'; -import { inspectUrl, listSitemaps, querySearchAnalytics, type InspectionResult } from './api'; +import { + inspectUrl, + listSitemaps, + querySearchAnalytics, + type InspectionFailure, + type InspectionResult, +} from './api'; const OUT_DIR = path.join(process.cwd(), 'apps', 'website', '.gsc'); @@ -29,11 +35,6 @@ async function sitemapUrls(): Promise { return urls; } -interface InspectionFailure { - url: string; - error: string; -} - async function main(): Promise { // Search Analytics data lags ~2 days; end 3 days back for a stable window. const endDate = isoDaysAgo(3); diff --git a/apps/website/scripts/gsc/report.ts b/apps/website/scripts/gsc/report.ts index 52167b2ea..34c4565d4 100644 --- a/apps/website/scripts/gsc/report.ts +++ b/apps/website/scripts/gsc/report.ts @@ -1,7 +1,7 @@ // SPDX-License-Identifier: MIT import fs from 'node:fs'; import path from 'node:path'; -import type { InspectionResult, SearchAnalyticsRow } from './api'; +import type { InspectionFailure, InspectionResult, SearchAnalyticsRow } from './api'; import { capList, describeInspectionCoverage, @@ -11,30 +11,12 @@ import { findWeakCtr, findZeroImpressionPages, } from './analysis'; +import { read, readOptional } from './snapshots'; const DIR = path.join(process.cwd(), 'apps', 'website', '.gsc'); -/** A URL Inspection call that failed during the pull, as recorded by pull.ts. */ -interface InspectionFailure { - url: string; - error: string; -} - -function read(name: string): T { - return JSON.parse(fs.readFileSync(path.join(DIR, name), 'utf8')) as T; -} - -/** Like `read`, for a file the pull only writes on a partial sweep. Absent is null; nothing else is swallowed. */ -function readOptional(name: string): T | null { - try { - return read(name); - } catch (error) { - if ((error as NodeJS.ErrnoException).code === 'ENOENT') { - return null; - } - throw error; - } -} +/** Longest any bullet list in the report gets before it is trimmed with a count. */ +const LIST_LIMIT = 20; function table(rows: SearchAnalyticsRow[], headers: string[], limit = 30): string { const head = `| ${headers.join(' | ')} |\n| ${headers.map(() => '---').join(' | ')} |`; @@ -48,12 +30,27 @@ function table(rows: SearchAnalyticsRow[], headers: string[], limit = 30): strin return `${head}\n${body}`; } +/** A `###` heading over a capped bullet list, or `_none_` when there is nothing to say. */ +function bulletSection(heading: string, items: string[]): string[] { + if (items.length === 0) { + return [`### ${heading}`, ``, `_none_`, ``]; + } + const { shown, remaining } = capList(items, LIST_LIMIT); + return [ + `### ${heading}`, + ``, + shown.map((item) => `- ${item}`).join('\n'), + ...(remaining > 0 ? [``, `_…and ${remaining} more._`] : []), + ``, + ]; +} + function main(): void { - const meta = read<{ startDate: string; endDate: string }>('meta.json'); - const queries = read('queries.json'); - const pages = read('pages.json'); - const inspections = read('inspections.json'); - const failures = readOptional('inspection-errors.json') ?? []; + const meta = read<{ startDate: string; endDate: string }>(DIR, 'meta.json'); + const queries = read(DIR, 'queries.json'); + const pages = read(DIR, 'pages.json'); + const inspections = read(DIR, 'inspections.json'); + const failures = readOptional(DIR, 'inspection-errors.json') ?? []; // Sitemap inventory = URLs we inspected PLUS URLs we failed to inspect, so a // failed URL still reaches the zero-impression analysis instead of vanishing. const sitemapUrls = [...inspections.map((i) => i.url), ...failures.map((f) => f.url)]; @@ -69,10 +66,6 @@ function main(): void { const unindexed = findUnindexed(inspections); const mismatches = findCanonicalMismatches(inspections); const orphans = findZeroImpressionPages(sitemapUrls, pages); - const failed = capList( - failures.map((failure) => `${failure.url} — ${failure.error}`), - 20, - ); const report = [ `# threadplane.ai — Search Console report`, @@ -92,29 +85,22 @@ function main(): void { `- Google canonical ≠ our canonical: ${mismatches.length}`, `- Zero-impression pages in window: ${orphans.length}`, ``, + // Only rendered on a partial sweep; a clean run should not carry an empty section. ...(failures.length > 0 - ? [ - `### Failed inspections`, - ``, - failed.shown.map((line) => `- ${line}`).join('\n'), - ...(failed.remaining > 0 ? [``, `_…and ${failed.remaining} more._`] : []), - ``, - ] + ? bulletSection( + 'Failed inspections', + failures.map((failure) => `${failure.url} — ${failure.error}`), + ) : []), - `### Not indexed`, - ``, - unindexed.length ? unindexed.map((i) => `- ${i.url} — ${i.coverageState}`).join('\n') : '_none_', - ``, - `### Canonical mismatches`, - ``, - mismatches.length - ? mismatches.map((i) => `- ${i.url} → Google chose ${i.googleCanonical}`).join('\n') - : '_none_', - ``, - `### Zero-impression pages`, - ``, - orphans.length ? orphans.map((url) => `- ${url}`).join('\n') : '_none_', - ``, + ...bulletSection( + 'Not indexed', + unindexed.map((i) => `${i.url} — ${i.coverageState}`), + ), + ...bulletSection( + 'Canonical mismatches', + mismatches.map((i) => `${i.url} → Google chose ${i.googleCanonical}`), + ), + ...bulletSection('Zero-impression pages', orphans), `## Striking distance (position 5–20, ≥50 impressions)`, ``, table(findStrikingDistance(queries, { minImpressions: 50 }), [ @@ -127,7 +113,7 @@ function main(): void { ``, `## Weak CTR on page one (≥100 impressions, CTR < 2%)`, ``, - `Title/description rewrite candidates.`, + `Title/description rewrite candidates. The threshold is flat across positions 1–10 — read the Pos column before acting.`, ``, table(findWeakCtr(queries, { minImpressions: 100, maxCtr: 0.02 }), [ 'Query', @@ -139,7 +125,7 @@ function main(): void { ``, `## Top pages`, ``, - table(pages.slice(0, 30), ['Page', 'Clicks', 'Impr', 'CTR', 'Pos']), + table(pages, ['Page', 'Clicks', 'Impr', 'CTR', 'Pos']), ``, ].join('\n'); @@ -147,4 +133,11 @@ function main(): void { console.log('wrote .gsc/report.md'); } -main(); +try { + main(); +} catch (error) { + // The likely failures here are "you have not run the pull yet" and "a snapshot + // is corrupt", both of which read better as one line than as a stack trace. + console.error(error instanceof Error ? error.message : String(error)); + process.exitCode = 1; +} diff --git a/apps/website/scripts/gsc/snapshots.spec.ts b/apps/website/scripts/gsc/snapshots.spec.ts new file mode 100644 index 000000000..886a60254 --- /dev/null +++ b/apps/website/scripts/gsc/snapshots.spec.ts @@ -0,0 +1,48 @@ +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import { read, readOptional } from './snapshots'; + +let dir: string; + +beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'gsc-snapshots-')); +}); + +afterEach(() => { + fs.rmSync(dir, { recursive: true, force: true }); +}); + +describe('read', () => { + it('parses a snapshot file', () => { + fs.writeFileSync(path.join(dir, 'meta.json'), '{"startDate":"2026-01-01"}'); + expect(read<{ startDate: string }>(dir, 'meta.json')).toEqual({ startDate: '2026-01-01' }); + }); + + it('names the missing file and points at the pull instead of throwing ENOENT', () => { + expect(() => read(dir, 'queries.json')).toThrow(/Missing snapshot .*queries\.json/); + expect(() => read(dir, 'queries.json')).toThrow(/npm run gsc:pull/); + }); + + it('reports malformed JSON with the offending file', () => { + fs.writeFileSync(path.join(dir, 'pages.json'), '{ broken'); + expect(() => read(dir, 'pages.json')).toThrow(/Malformed JSON in .*pages\.json/); + }); +}); + +describe('readOptional', () => { + it('returns null when the file is genuinely absent', () => { + expect(readOptional(dir, 'inspection-errors.json')).toBeNull(); + }); + + it('still rethrows on malformed JSON rather than reporting absence', () => { + fs.writeFileSync(path.join(dir, 'inspection-errors.json'), '{ broken'); + expect(() => readOptional(dir, 'inspection-errors.json')).toThrow(/Malformed JSON/); + }); + + it('parses the file when it is present', () => { + fs.writeFileSync(path.join(dir, 'inspection-errors.json'), '[{"url":"u","error":"429"}]'); + expect(readOptional(dir, 'inspection-errors.json')).toEqual([{ url: 'u', error: '429' }]); + }); +}); diff --git a/apps/website/scripts/gsc/snapshots.ts b/apps/website/scripts/gsc/snapshots.ts new file mode 100644 index 000000000..9b181d6da --- /dev/null +++ b/apps/website/scripts/gsc/snapshots.ts @@ -0,0 +1,41 @@ +// SPDX-License-Identifier: MIT +import fs from 'node:fs'; +import path from 'node:path'; + +/** + * Read one snapshot file written by `pull.ts`. The overwhelmingly likely error + * is running the report before the pull, so say that in the message instead of + * surfacing a raw ENOENT stack. + */ +export function read(dir: string, name: string): T { + const file = path.join(dir, name); + let raw: string; + try { + raw = fs.readFileSync(file, 'utf8'); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === 'ENOENT') { + throw new Error(`Missing snapshot ${file}. Run \`npm run gsc:pull\` first.`); + } + throw error; + } + try { + return JSON.parse(raw) as T; + } catch (error) { + throw new Error( + `Malformed JSON in ${file}: ${(error as Error).message}. Re-run \`npm run gsc:pull\`.`, + ); + } +} + +/** + * Like {@link read}, for a file `pull.ts` writes only on a partial sweep. + * Genuine absence is `null`; every other failure — malformed JSON, permissions — + * still throws. The existence check is deliberate: `read` now reports a missing + * file as a friendly Error, so `.code` is no longer available to discriminate on. + */ +export function readOptional(dir: string, name: string): T | null { + if (!fs.existsSync(path.join(dir, name))) { + return null; + } + return read(dir, name); +} From 68f1eece2bcfabd2b41fb317082dc75e346794d2 Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 20 Aug 2026 12:14:09 -0700 Subject: [PATCH 09/35] docs(plan): correct website test command (no nx test target exists) Co-Authored-By: Claude Opus 5 --- .../2026-08-20-ai-search-optimization.md | 24 +++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/superpowers/plans/2026-08-20-ai-search-optimization.md b/docs/superpowers/plans/2026-08-20-ai-search-optimization.md index 3d56fc8d9..2662b1df6 100644 --- a/docs/superpowers/plans/2026-08-20-ai-search-optimization.md +++ b/docs/superpowers/plans/2026-08-20-ai-search-optimization.md @@ -522,7 +522,7 @@ describe('findUnindexed', () => { - [ ] **Step 3: Run it and confirm it fails** ```bash -npx nx test website +cd apps/website && npx vitest run --config vite.config.mts ``` Expected: FAIL — `Failed to resolve import "./analysis"`. @@ -593,7 +593,7 @@ export function findWeakCtr( - [ ] **Step 5: Run the tests and confirm they pass** ```bash -npx nx test website +cd apps/website && npx vitest run --config vite.config.mts ``` Expected: PASS. @@ -767,7 +767,7 @@ describe('getSitemapEntries', () => { - [ ] **Step 2: Run it and confirm it fails** ```bash -npx nx test website +cd apps/website && npx vitest run --config vite.config.mts ``` Expected: FAIL — `getSitemapEntries is not a function`. @@ -839,7 +839,7 @@ export default function sitemap(): MetadataRoute.Sitemap { - [ ] **Step 5: Run the tests and confirm they pass** ```bash -npx nx test website +cd apps/website && npx vitest run --config vite.config.mts ``` Expected: PASS. @@ -921,7 +921,7 @@ Add `SITE_NAME` to the existing import list at the top of that spec file. - [ ] **Step 2: Run it and confirm it fails** ```bash -npx nx test website +cd apps/website && npx vitest run --config vite.config.mts ``` Expected: FAIL — `publishedTime` is `undefined`. @@ -1043,7 +1043,7 @@ Expected: `0`. - [ ] **Step 7: Run the tests and confirm they pass** ```bash -npx nx test website +cd apps/website && npx vitest run --config vite.config.mts ``` Expected: PASS. @@ -1150,7 +1150,7 @@ describe('softwareSourceCodeJsonLd', () => { - [ ] **Step 2: Run it and confirm it fails** ```bash -npx nx test website +cd apps/website && npx vitest run --config vite.config.mts ``` Expected: FAIL — `Failed to resolve import "./structured-data"`. @@ -1281,7 +1281,7 @@ export function faqJsonLd(entries: { question: string; answer: string }[]): Json - [ ] **Step 4: Run the tests and confirm they pass** ```bash -npx nx test website +cd apps/website && npx vitest run --config vite.config.mts ``` Expected: PASS. If `organizationJsonLd` fails on the logo path, run `ls apps/website/public/logos` and use the actual filename. @@ -1684,7 +1684,7 @@ In `apps/website/src/components/shared/Footer.tsx`, add an `About` link to `/abo - [ ] **Step 4: Verify** ```bash -npx nx test website && npx nx build website +cd apps/website && npx vitest run --config vite.config.mts && npx nx build website ``` Expected: PASS and a successful build. Then serve and confirm `curl -s http://localhost:3000/about | grep -c 'application/ld+json'` returns `2`. @@ -1890,7 +1890,7 @@ describe('classifyAiReferrer', () => { - [ ] **Step 2: Run it and confirm it fails** ```bash -npx nx test website +cd apps/website && npx vitest run --config vite.config.mts ``` Expected: FAIL — `Failed to resolve import "./ai-traffic"`. @@ -1957,7 +1957,7 @@ export function classifyAiReferrer(referrer: string): string | null { - [ ] **Step 4: Run the tests and confirm they pass** ```bash -npx nx test website +cd apps/website && npx vitest run --config vite.config.mts ``` Expected: PASS. @@ -2134,7 +2134,7 @@ git commit -m "docs(gtm): ai search measurement runbook" - [ ] **Step 1: Full check** ```bash -npx nx test website && npx nx lint website && npx nx build website --configuration=production +cd apps/website && npx vitest run --config vite.config.mts && npx nx lint website && npx nx build website --configuration=production ``` Expected: all three pass. Lint **warnings** are tolerated by CI; lint **errors** are not. To count errors, strip ANSI first — `grep -cE ' error '` on raw output silently returns 0: From 2c435fc209169a36447742628a946b6b4b556ab0 Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 20 Aug 2026 12:15:34 -0700 Subject: [PATCH 10/35] test(website): fix stale positioning proof-point assertion --- apps/website/src/lib/site-metadata.spec.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/apps/website/src/lib/site-metadata.spec.ts b/apps/website/src/lib/site-metadata.spec.ts index ffdb37680..5e9db745d 100644 --- a/apps/website/src/lib/site-metadata.spec.ts +++ b/apps/website/src/lib/site-metadata.spec.ts @@ -25,7 +25,7 @@ describe('site positioning copy', () => { 'json-render + A2UI', ]); expect(POSITIONING_PROOF_POINTS.map((p) => p.href)).toEqual([ - '/docs/langgraph/concepts/langgraph-basics', + '/docs/choosing-an-adapter', '/docs/langgraph/guides/persistence', '/docs/langgraph/guides/interrupts', '/docs/langgraph/guides/subgraphs', From 87a7979d8bb7fdcf23e77c0f59da2a2bb370ce67 Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 20 Aug 2026 12:24:10 -0700 Subject: [PATCH 11/35] feat(website): emit honest lastmod in sitemap Google ignores changefreq/priority and uses lastmod when it is honest, so the sitemap now emits only lastmod, derived per route from its real source: blog frontmatter dates, docs .mdx files, and page.tsx templates (plus the solutions data module for the programmatic /solutions/* pages). Times come from git commit history rather than file mtimes: a fresh CI checkout rewrites every mtime to clone time, which would claim the whole site changed on every deploy. Shallow clones are detected and their grafted boundary commits discarded, and any route we cannot date honestly simply omits lastmod rather than fabricating one. Co-Authored-By: Claude Opus 5 --- apps/website/src/app/sitemap.ts | 11 +- apps/website/src/lib/site-metadata.spec.ts | 35 ++++ apps/website/src/lib/site-metadata.ts | 192 +++++++++++++++++++++ 3 files changed, 233 insertions(+), 5 deletions(-) diff --git a/apps/website/src/app/sitemap.ts b/apps/website/src/app/sitemap.ts index 3165aabbf..832172709 100644 --- a/apps/website/src/app/sitemap.ts +++ b/apps/website/src/app/sitemap.ts @@ -1,10 +1,11 @@ import type { MetadataRoute } from 'next'; -import { getCanonicalUrl, getSitemapRoutes } from '../lib/site-metadata'; +import { getCanonicalUrl, getSitemapEntries } from '../lib/site-metadata'; +// `changefreq`/`priority` are ignored by Google; `lastmod` is used when it is +// honest, so this emits only that. export default function sitemap(): MetadataRoute.Sitemap { - return getSitemapRoutes().map((route) => ({ - url: getCanonicalUrl(route), - changeFrequency: route.startsWith('/docs') ? 'weekly' : 'monthly', - priority: route === '/' ? 1 : route.startsWith('/docs') ? 0.8 : 0.7, + return getSitemapEntries().map((entry) => ({ + url: getCanonicalUrl(entry.route), + ...(entry.lastModified ? { lastModified: entry.lastModified } : {}), })); } diff --git a/apps/website/src/lib/site-metadata.spec.ts b/apps/website/src/lib/site-metadata.spec.ts index 5e9db745d..15380cb98 100644 --- a/apps/website/src/lib/site-metadata.spec.ts +++ b/apps/website/src/lib/site-metadata.spec.ts @@ -49,3 +49,38 @@ describe('site positioning copy', () => { expect(metadata.twitter?.description).toBe(DEFAULT_META_DESCRIPTION); }); }); + +describe('getSitemapEntries', () => { + it('emits a valid lastModified date for every route', async () => { + const { getSitemapEntries } = await import('./site-metadata'); + const entries = getSitemapEntries(); + expect(entries.length).toBeGreaterThan(100); + const unresolved = entries.filter( + (e) => !(e.lastModified instanceof Date) || Number.isNaN(e.lastModified.getTime()), + ); + expect(unresolved.map((e) => e.route)).toEqual([]); + }); + + it('uses the post date as lastModified for blog routes', async () => { + const { getSitemapEntries } = await import('./site-metadata'); + const entry = getSitemapEntries().find((e) => e.route === '/blog/angular-chat-app-tutorial-with-ag-ui'); + expect(entry?.lastModified?.toISOString().slice(0, 10)).toBe('2026-08-13'); + }); + + it('derives distinct, non-"now" dates rather than stamping the whole site with build time', async () => { + const { getSitemapEntries } = await import('./site-metadata'); + const entries = getSitemapEntries(); + const distinct = new Set(entries.map((e) => e.lastModified?.toISOString().slice(0, 10))); + expect(distinct.size).toBeGreaterThan(5); + + const docsEntry = entries.find((e) => e.route === '/docs/langgraph/getting-started/introduction'); + expect(docsEntry?.lastModified).toBeInstanceOf(Date); + expect(docsEntry?.lastModified?.getTime()).toBeLessThan(Date.now()); + }); + + it('resolves the special docs pages whose route shape differs from library docs', async () => { + const { getSitemapEntries } = await import('./site-metadata'); + const entry = getSitemapEntries().find((e) => e.route === '/docs/choosing-an-adapter'); + expect(entry?.lastModified).toBeInstanceOf(Date); + }); +}); diff --git a/apps/website/src/lib/site-metadata.ts b/apps/website/src/lib/site-metadata.ts index 6a5744005..b7d724d26 100644 --- a/apps/website/src/lib/site-metadata.ts +++ b/apps/website/src/lib/site-metadata.ts @@ -1,3 +1,6 @@ +import { execFileSync } from 'node:child_process'; +import fs from 'node:fs'; +import path from 'node:path'; import type { Metadata } from 'next'; import { getAllSolutionSlugs } from './solutions-data'; import { docsConfig, specialDocsPages } from './docs-config'; @@ -74,3 +77,192 @@ export function getSitemapRoutes(): string[] { return [...staticRoutes, ...solutionRoutes, ...docsRoutes, ...specialDocsRoutes, ...blogRoutes]; } + +/** A sitemap URL plus the honest last-modified time of the source it renders from. */ +export interface SitemapEntry { + route: string; + /** + * Omitted when no honest value can be determined. An absent `` is + * valid sitemap XML; a fabricated one (e.g. "now" on every build, which is + * what raw file mtimes give you on a fresh CI checkout) teaches crawlers to + * ignore the signal across the whole site. + */ + lastModified?: Date; +} + +/** Website-relative source paths that a route's content is rendered from. */ +function sourcePathsForRoute(route: string): string[] { + const special = specialDocsPages.find((page) => page.path === route); + if (special) return [path.join('content', 'docs', special.contentPath)]; + + if (route.startsWith('/docs/')) { + const [, , library, section, slug] = route.split('/'); + if (library && section && slug) { + return [path.join('content', 'docs', library, section, `${slug}.mdx`)]; + } + // Anything else under /docs is a hand-written route, dated like any other. + } + + if (route.startsWith('/solutions/')) { + // Programmatic pages: one dynamic route template rendering data from a + // single module, so a change to either re-renders the page. + return [ + path.join('src', 'app', 'solutions', '[slug]', 'page.tsx'), + path.join('src', 'lib', 'solutions-data.ts'), + ]; + } + + const routeDir = route === '/' ? '' : route.replace(/^\//, ''); + return [path.join('src', 'app', routeDir, 'page.tsx')]; +} + +// `nx build website` and a standalone `next build` disagree about cwd, exactly +// as `blog.ts` already has to handle. +const WEBSITE_DIR_CANDIDATES = [path.join(process.cwd(), 'apps', 'website'), process.cwd()]; + +/** Absolute path for a website-relative source path, or null when it is missing. */ +function resolveSourcePath(relativePath: string): string | null { + for (const dir of WEBSITE_DIR_CANDIDATES) { + const candidate = path.join(dir, relativePath); + if (fs.existsSync(candidate)) return candidate; + } + return null; +} + +const GIT_LOG_MARKER = 'commit-time '; + +interface GitTimes { + /** Last commit time (epoch seconds) keyed by website-relative path. */ + times: Map; + /** + * True when the clone has full history, which is also the only situation in + * which a file's absence from `times` means "never committed" (and so its + * mtime is a real edit time rather than a checkout timestamp). + */ + complete: boolean; +} + +let gitTimesCache: GitTimes | null | undefined; + +/** + * Last commit time (epoch seconds) keyed by website-relative path. + * + * Returns null when git history yields nothing usable (no git, not a repo, or a + * clone so shallow that only grafted commits are visible). + * + * Shallow clones — which is what CI hosts do by default — need care: a grafted + * boundary commit has no parent, so `git log --name-only` reports *every* file + * in the tree as changed at that commit's timestamp. Those entries are dropped; + * files git genuinely saw change in the visible history keep real dates, and + * everything older simply gets no `lastmod`. + */ +function computeGitTimes(): GitTimes | null { + try { + const git = (args: string[], cwd: string): string => + execFileSync('git', args, { + cwd, + encoding: 'utf8', + maxBuffer: 64 * 1024 * 1024, + stdio: ['ignore', 'pipe', 'ignore'], + }); + + const root = git(['rev-parse', '--show-toplevel'], process.cwd()).trim(); + const graftedCommits = new Set(); + if (git(['rev-parse', '--is-shallow-repository'], root).trim() === 'true') { + const shallowFile = path.join(git(['rev-parse', '--absolute-git-dir'], root).trim(), 'shallow'); + if (!fs.existsSync(shallowFile)) return null; + for (const sha of fs.readFileSync(shallowFile, 'utf8').split('\n')) { + if (sha.trim()) graftedCommits.add(sha.trim()); + } + } + + const prefix = fs.existsSync(path.join(root, 'apps', 'website')) ? 'apps/website/' : ''; + const log = git( + [ + 'log', + `--format=${GIT_LOG_MARKER}%ct %H`, + '--name-only', + '--', + `${prefix}content`, + `${prefix}src/app`, + `${prefix}src/lib`, + ], + root, + ); + + const times = new Map(); + let commitTime = 0; + for (const line of log.split('\n')) { + if (line.startsWith(GIT_LOG_MARKER)) { + const [seconds, sha] = line.slice(GIT_LOG_MARKER.length).split(' '); + commitTime = graftedCommits.has(sha) ? 0 : Number(seconds); + continue; + } + const file = line.trim(); + if (!file || !commitTime) continue; + const relative = prefix && file.startsWith(prefix) ? file.slice(prefix.length) : file; + // `git log` walks newest-first, so the first mention of a path wins. + if (!times.has(relative)) times.set(relative, commitTime); + } + return times.size > 0 ? { times, complete: graftedCommits.size === 0 } : null; + } catch { + return null; + } +} + +function getGitTimes(): GitTimes | null { + if (gitTimesCache !== undefined) return gitTimesCache; + gitTimesCache = computeGitTimes(); + + if (gitTimesCache === null) { + // Visible in the build log: the sitemap will carry `lastmod` only for blog + // posts (which date themselves in frontmatter) until history is available. + console.warn('[sitemap] git history unavailable; omitting for file-derived routes.'); + } + + return gitTimesCache; +} + +/** + * Newest honest modification time across a route's sources. + * + * Git commit time is authoritative. A file mtime is only trusted when the clone + * has full history and git still has no record of the file — i.e. uncommitted + * local work. Without that guard every URL would carry the checkout time, since + * a fresh CI clone rewrites every mtime to the moment it ran. + */ +function sourceModifiedTime(relativePaths: string[]): Date | undefined { + const git = getGitTimes(); + let newest: number | undefined; + + for (const relativePath of relativePaths) { + const absolute = resolveSourcePath(relativePath); + if (!absolute) continue; + + const gitSeconds = git?.times.get(relativePath.split(path.sep).join('/')); + const millis = + gitSeconds !== undefined + ? gitSeconds * 1000 + : git?.complete + ? fs.statSync(absolute).mtimeMs + : undefined; + if (millis === undefined) continue; + if (newest === undefined || millis > newest) newest = millis; + } + + return newest === undefined ? undefined : new Date(newest); +} + +export function getSitemapEntries(): SitemapEntry[] { + const blogDates = new Map( + getAllPosts().map((post) => [`/blog/${post.slug}`, new Date(`${post.frontmatter.date}T00:00:00Z`)]), + ); + + return getSitemapRoutes().map((route) => { + const blogDate = blogDates.get(route); + if (blogDate && !Number.isNaN(blogDate.getTime())) return { route, lastModified: blogDate }; + + const lastModified = sourceModifiedTime(sourcePathsForRoute(route)); + return lastModified ? { route, lastModified } : { route }; + }); +} From 160f84e4a865bc21d0a4da84d0bdffc97ac25fde Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 20 Aug 2026 12:32:24 -0700 Subject: [PATCH 12/35] fix(website): detect shallow clones through the common git dir The `shallow` marker lives in the common git dir, not the per-worktree gitdir, so `--absolute-git-dir` misses it inside a linked worktree (.git/worktrees/ vs .git) and the sitemap silently dropped lastmod for every file-derived route. Co-Authored-By: Claude Opus 5 --- apps/website/src/lib/site-metadata.ts | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/apps/website/src/lib/site-metadata.ts b/apps/website/src/lib/site-metadata.ts index b7d724d26..ac1fdc60d 100644 --- a/apps/website/src/lib/site-metadata.ts +++ b/apps/website/src/lib/site-metadata.ts @@ -169,7 +169,12 @@ function computeGitTimes(): GitTimes | null { const root = git(['rev-parse', '--show-toplevel'], process.cwd()).trim(); const graftedCommits = new Set(); if (git(['rev-parse', '--is-shallow-repository'], root).trim() === 'true') { - const shallowFile = path.join(git(['rev-parse', '--absolute-git-dir'], root).trim(), 'shallow'); + // Must be the COMMON git dir, not `--absolute-git-dir`: in a linked + // worktree those differ (`.git/worktrees/` vs `.git`) and `shallow` + // only ever lives in the common dir. Output can be relative, and git runs + // with cwd=root, so resolve it against root. + const commonDir = path.resolve(root, git(['rev-parse', '--git-common-dir'], root).trim()); + const shallowFile = path.join(commonDir, 'shallow'); if (!fs.existsSync(shallowFile)) return null; for (const sha of fs.readFileSync(shallowFile, 'utf8').split('\n')) { if (sha.trim()) graftedCommits.add(sha.trim()); From 207e49a928cb2b9be1374f4b57716c018b13d05b Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 20 Aug 2026 12:40:39 -0700 Subject: [PATCH 13/35] refactor(website): extract sitemap dates and close date-fabrication paths MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Moves the sitemap date logic out of site-metadata.ts into sitemap-dates.ts and removes the remaining ways a date could be invented: - Delete the file-mtime fallback. "Not shallow" never implied "committed" — a fresh full clone also rewrites every mtime to checkout time, so a single lookup miss could publish a build-time lastmod. - Run git with core.quotePath=false (octal-escaped non-ASCII paths would never match a lookup, silently falling through) and log.showSignature=false (a user's config could interleave gpg: lines into the file list). - Bound the subprocess with timeout/SIGKILL so a stalled git cannot hang the build; a kill throws and degrades like any other git failure. - Require a well-formed " " remainder before treating a line as a commit header, and take paths verbatim so leading whitespace survives. Blog routes now take the later of the frontmatter date and their .mdx commit time, since lastmod means last modified rather than published. Extracts parseGitLog as a pure function and unit-tests the degradation the design rests on — most importantly that a grafted shallow-boundary commit yields no entry rather than a clone-time one — plus a test pinning the absence of changeFrequency/priority. Integration tests now tolerate a history-less checkout instead of asserting completeness unconditionally. Co-Authored-By: Claude Opus 5 --- apps/website/src/app/sitemap.ts | 3 +- apps/website/src/lib/blog.ts | 8 +- apps/website/src/lib/site-metadata.spec.ts | 35 --- apps/website/src/lib/site-metadata.ts | 197 ----------------- apps/website/src/lib/sitemap-dates.spec.ts | 139 ++++++++++++ apps/website/src/lib/sitemap-dates.ts | 242 +++++++++++++++++++++ apps/website/src/lib/website-dir.ts | 12 + 7 files changed, 397 insertions(+), 239 deletions(-) create mode 100644 apps/website/src/lib/sitemap-dates.spec.ts create mode 100644 apps/website/src/lib/sitemap-dates.ts create mode 100644 apps/website/src/lib/website-dir.ts diff --git a/apps/website/src/app/sitemap.ts b/apps/website/src/app/sitemap.ts index 832172709..b1c73eb00 100644 --- a/apps/website/src/app/sitemap.ts +++ b/apps/website/src/app/sitemap.ts @@ -1,5 +1,6 @@ import type { MetadataRoute } from 'next'; -import { getCanonicalUrl, getSitemapEntries } from '../lib/site-metadata'; +import { getCanonicalUrl } from '../lib/site-metadata'; +import { getSitemapEntries } from '../lib/sitemap-dates'; // `changefreq`/`priority` are ignored by Google; `lastmod` is used when it is // honest, so this emits only that. diff --git a/apps/website/src/lib/blog.ts b/apps/website/src/lib/blog.ts index 7c090e85e..8ad405ae6 100644 --- a/apps/website/src/lib/blog.ts +++ b/apps/website/src/lib/blog.ts @@ -2,9 +2,7 @@ import fs from 'fs'; import path from 'path'; import matter from 'gray-matter'; - -const BLOG_DIR_WORKSPACE = path.join(process.cwd(), 'apps', 'website', 'content', 'blog'); -const BLOG_DIR_LOCAL = path.join(process.cwd(), 'content', 'blog'); +import { resolveWebsiteDir } from './website-dir'; export interface PostFrontmatter { title: string; @@ -27,9 +25,7 @@ export interface Post { const FILENAME_RE = /^(\d{4}-\d{2}-\d{2})-(.+)\.mdx$/; function resolveBlogDir(): string { - if (fs.existsSync(BLOG_DIR_WORKSPACE)) return BLOG_DIR_WORKSPACE; - if (fs.existsSync(BLOG_DIR_LOCAL)) return BLOG_DIR_LOCAL; - return BLOG_DIR_WORKSPACE; + return path.join(resolveWebsiteDir(), 'content', 'blog'); } function readPost(dir: string, filename: string): Post | null { diff --git a/apps/website/src/lib/site-metadata.spec.ts b/apps/website/src/lib/site-metadata.spec.ts index 15380cb98..5e9db745d 100644 --- a/apps/website/src/lib/site-metadata.spec.ts +++ b/apps/website/src/lib/site-metadata.spec.ts @@ -49,38 +49,3 @@ describe('site positioning copy', () => { expect(metadata.twitter?.description).toBe(DEFAULT_META_DESCRIPTION); }); }); - -describe('getSitemapEntries', () => { - it('emits a valid lastModified date for every route', async () => { - const { getSitemapEntries } = await import('./site-metadata'); - const entries = getSitemapEntries(); - expect(entries.length).toBeGreaterThan(100); - const unresolved = entries.filter( - (e) => !(e.lastModified instanceof Date) || Number.isNaN(e.lastModified.getTime()), - ); - expect(unresolved.map((e) => e.route)).toEqual([]); - }); - - it('uses the post date as lastModified for blog routes', async () => { - const { getSitemapEntries } = await import('./site-metadata'); - const entry = getSitemapEntries().find((e) => e.route === '/blog/angular-chat-app-tutorial-with-ag-ui'); - expect(entry?.lastModified?.toISOString().slice(0, 10)).toBe('2026-08-13'); - }); - - it('derives distinct, non-"now" dates rather than stamping the whole site with build time', async () => { - const { getSitemapEntries } = await import('./site-metadata'); - const entries = getSitemapEntries(); - const distinct = new Set(entries.map((e) => e.lastModified?.toISOString().slice(0, 10))); - expect(distinct.size).toBeGreaterThan(5); - - const docsEntry = entries.find((e) => e.route === '/docs/langgraph/getting-started/introduction'); - expect(docsEntry?.lastModified).toBeInstanceOf(Date); - expect(docsEntry?.lastModified?.getTime()).toBeLessThan(Date.now()); - }); - - it('resolves the special docs pages whose route shape differs from library docs', async () => { - const { getSitemapEntries } = await import('./site-metadata'); - const entry = getSitemapEntries().find((e) => e.route === '/docs/choosing-an-adapter'); - expect(entry?.lastModified).toBeInstanceOf(Date); - }); -}); diff --git a/apps/website/src/lib/site-metadata.ts b/apps/website/src/lib/site-metadata.ts index ac1fdc60d..6a5744005 100644 --- a/apps/website/src/lib/site-metadata.ts +++ b/apps/website/src/lib/site-metadata.ts @@ -1,6 +1,3 @@ -import { execFileSync } from 'node:child_process'; -import fs from 'node:fs'; -import path from 'node:path'; import type { Metadata } from 'next'; import { getAllSolutionSlugs } from './solutions-data'; import { docsConfig, specialDocsPages } from './docs-config'; @@ -77,197 +74,3 @@ export function getSitemapRoutes(): string[] { return [...staticRoutes, ...solutionRoutes, ...docsRoutes, ...specialDocsRoutes, ...blogRoutes]; } - -/** A sitemap URL plus the honest last-modified time of the source it renders from. */ -export interface SitemapEntry { - route: string; - /** - * Omitted when no honest value can be determined. An absent `` is - * valid sitemap XML; a fabricated one (e.g. "now" on every build, which is - * what raw file mtimes give you on a fresh CI checkout) teaches crawlers to - * ignore the signal across the whole site. - */ - lastModified?: Date; -} - -/** Website-relative source paths that a route's content is rendered from. */ -function sourcePathsForRoute(route: string): string[] { - const special = specialDocsPages.find((page) => page.path === route); - if (special) return [path.join('content', 'docs', special.contentPath)]; - - if (route.startsWith('/docs/')) { - const [, , library, section, slug] = route.split('/'); - if (library && section && slug) { - return [path.join('content', 'docs', library, section, `${slug}.mdx`)]; - } - // Anything else under /docs is a hand-written route, dated like any other. - } - - if (route.startsWith('/solutions/')) { - // Programmatic pages: one dynamic route template rendering data from a - // single module, so a change to either re-renders the page. - return [ - path.join('src', 'app', 'solutions', '[slug]', 'page.tsx'), - path.join('src', 'lib', 'solutions-data.ts'), - ]; - } - - const routeDir = route === '/' ? '' : route.replace(/^\//, ''); - return [path.join('src', 'app', routeDir, 'page.tsx')]; -} - -// `nx build website` and a standalone `next build` disagree about cwd, exactly -// as `blog.ts` already has to handle. -const WEBSITE_DIR_CANDIDATES = [path.join(process.cwd(), 'apps', 'website'), process.cwd()]; - -/** Absolute path for a website-relative source path, or null when it is missing. */ -function resolveSourcePath(relativePath: string): string | null { - for (const dir of WEBSITE_DIR_CANDIDATES) { - const candidate = path.join(dir, relativePath); - if (fs.existsSync(candidate)) return candidate; - } - return null; -} - -const GIT_LOG_MARKER = 'commit-time '; - -interface GitTimes { - /** Last commit time (epoch seconds) keyed by website-relative path. */ - times: Map; - /** - * True when the clone has full history, which is also the only situation in - * which a file's absence from `times` means "never committed" (and so its - * mtime is a real edit time rather than a checkout timestamp). - */ - complete: boolean; -} - -let gitTimesCache: GitTimes | null | undefined; - -/** - * Last commit time (epoch seconds) keyed by website-relative path. - * - * Returns null when git history yields nothing usable (no git, not a repo, or a - * clone so shallow that only grafted commits are visible). - * - * Shallow clones — which is what CI hosts do by default — need care: a grafted - * boundary commit has no parent, so `git log --name-only` reports *every* file - * in the tree as changed at that commit's timestamp. Those entries are dropped; - * files git genuinely saw change in the visible history keep real dates, and - * everything older simply gets no `lastmod`. - */ -function computeGitTimes(): GitTimes | null { - try { - const git = (args: string[], cwd: string): string => - execFileSync('git', args, { - cwd, - encoding: 'utf8', - maxBuffer: 64 * 1024 * 1024, - stdio: ['ignore', 'pipe', 'ignore'], - }); - - const root = git(['rev-parse', '--show-toplevel'], process.cwd()).trim(); - const graftedCommits = new Set(); - if (git(['rev-parse', '--is-shallow-repository'], root).trim() === 'true') { - // Must be the COMMON git dir, not `--absolute-git-dir`: in a linked - // worktree those differ (`.git/worktrees/` vs `.git`) and `shallow` - // only ever lives in the common dir. Output can be relative, and git runs - // with cwd=root, so resolve it against root. - const commonDir = path.resolve(root, git(['rev-parse', '--git-common-dir'], root).trim()); - const shallowFile = path.join(commonDir, 'shallow'); - if (!fs.existsSync(shallowFile)) return null; - for (const sha of fs.readFileSync(shallowFile, 'utf8').split('\n')) { - if (sha.trim()) graftedCommits.add(sha.trim()); - } - } - - const prefix = fs.existsSync(path.join(root, 'apps', 'website')) ? 'apps/website/' : ''; - const log = git( - [ - 'log', - `--format=${GIT_LOG_MARKER}%ct %H`, - '--name-only', - '--', - `${prefix}content`, - `${prefix}src/app`, - `${prefix}src/lib`, - ], - root, - ); - - const times = new Map(); - let commitTime = 0; - for (const line of log.split('\n')) { - if (line.startsWith(GIT_LOG_MARKER)) { - const [seconds, sha] = line.slice(GIT_LOG_MARKER.length).split(' '); - commitTime = graftedCommits.has(sha) ? 0 : Number(seconds); - continue; - } - const file = line.trim(); - if (!file || !commitTime) continue; - const relative = prefix && file.startsWith(prefix) ? file.slice(prefix.length) : file; - // `git log` walks newest-first, so the first mention of a path wins. - if (!times.has(relative)) times.set(relative, commitTime); - } - return times.size > 0 ? { times, complete: graftedCommits.size === 0 } : null; - } catch { - return null; - } -} - -function getGitTimes(): GitTimes | null { - if (gitTimesCache !== undefined) return gitTimesCache; - gitTimesCache = computeGitTimes(); - - if (gitTimesCache === null) { - // Visible in the build log: the sitemap will carry `lastmod` only for blog - // posts (which date themselves in frontmatter) until history is available. - console.warn('[sitemap] git history unavailable; omitting for file-derived routes.'); - } - - return gitTimesCache; -} - -/** - * Newest honest modification time across a route's sources. - * - * Git commit time is authoritative. A file mtime is only trusted when the clone - * has full history and git still has no record of the file — i.e. uncommitted - * local work. Without that guard every URL would carry the checkout time, since - * a fresh CI clone rewrites every mtime to the moment it ran. - */ -function sourceModifiedTime(relativePaths: string[]): Date | undefined { - const git = getGitTimes(); - let newest: number | undefined; - - for (const relativePath of relativePaths) { - const absolute = resolveSourcePath(relativePath); - if (!absolute) continue; - - const gitSeconds = git?.times.get(relativePath.split(path.sep).join('/')); - const millis = - gitSeconds !== undefined - ? gitSeconds * 1000 - : git?.complete - ? fs.statSync(absolute).mtimeMs - : undefined; - if (millis === undefined) continue; - if (newest === undefined || millis > newest) newest = millis; - } - - return newest === undefined ? undefined : new Date(newest); -} - -export function getSitemapEntries(): SitemapEntry[] { - const blogDates = new Map( - getAllPosts().map((post) => [`/blog/${post.slug}`, new Date(`${post.frontmatter.date}T00:00:00Z`)]), - ); - - return getSitemapRoutes().map((route) => { - const blogDate = blogDates.get(route); - if (blogDate && !Number.isNaN(blogDate.getTime())) return { route, lastModified: blogDate }; - - const lastModified = sourceModifiedTime(sourcePathsForRoute(route)); - return lastModified ? { route, lastModified } : { route }; - }); -} diff --git a/apps/website/src/lib/sitemap-dates.spec.ts b/apps/website/src/lib/sitemap-dates.spec.ts new file mode 100644 index 000000000..bdc7f2eb2 --- /dev/null +++ b/apps/website/src/lib/sitemap-dates.spec.ts @@ -0,0 +1,139 @@ +import { describe, expect, it } from 'vitest'; +import sitemap from '../app/sitemap'; +import { getSitemapEntries, hasGitHistory, parseGitLog } from './sitemap-dates'; + +const SHA_A = 'a'.repeat(40); +const SHA_B = 'b'.repeat(40); +const GRAFT = 'c'.repeat(40); + +function header(seconds: number, sha: string): string { + return `commit-time ${seconds} ${sha}`; +} + +describe('parseGitLog', () => { + it('takes the newest commit that touched each path', () => { + const log = [ + header(3000, SHA_A), + 'apps/website/content/docs/a.mdx', + '', + header(1000, SHA_B), + 'apps/website/content/docs/a.mdx', + 'apps/website/content/docs/b.mdx', + '', + ].join('\n'); + + const times = parseGitLog(log, 'apps/website/', new Set()); + + expect(Object.fromEntries(times)).toEqual({ 'content/docs/a.mdx': 3000, 'content/docs/b.mdx': 1000 }); + }); + + it('drops every file a grafted shallow-boundary commit claims to have changed', () => { + const log = [header(9000, GRAFT), 'content/docs/a.mdx', 'content/docs/b.mdx', ''].join('\n'); + + const times = parseGitLog(log, '', new Set([GRAFT])); + + // Not "dated 9000" — a graft reports the whole tree, so those dates are + // clone time, not change time, and must not be emitted at all. + expect(times.size).toBe(0); + }); + + it('keeps real commits in a stream that also contains a graft', () => { + const log = [ + header(4000, SHA_A), + 'content/docs/a.mdx', + '', + header(9000, GRAFT), + 'content/docs/a.mdx', + 'content/docs/b.mdx', + '', + ].join('\n'); + + const times = parseGitLog(log, '', new Set([GRAFT])); + + expect(Object.fromEntries(times)).toEqual({ 'content/docs/a.mdx': 4000 }); + }); + + it('handles a merge commit that lists no files', () => { + const log = [header(5000, SHA_A), '', header(4000, SHA_B), 'content/docs/a.mdx', ''].join('\n'); + + expect(Object.fromEntries(parseGitLog(log, '', new Set()))).toEqual({ 'content/docs/a.mdx': 4000 }); + }); + + it('takes paths verbatim, including non-ASCII and leading whitespace', () => { + const log = [header(7000, SHA_A), 'content/blog/café niño.mdx', ' leading-space.mdx', ''].join('\n'); + + expect(Object.fromEntries(parseGitLog(log, '', new Set()))).toEqual({ + 'content/blog/café niño.mdx': 7000, + ' leading-space.mdx': 7000, + }); + }); + + it('treats a path that begins with the marker text as a path, not a header', () => { + const log = [header(8000, SHA_A), 'commit-time notes.mdx', 'commit-time 123 nonsense', ''].join('\n'); + + expect(Object.fromEntries(parseGitLog(log, '', new Set()))).toEqual({ + 'commit-time notes.mdx': 8000, + 'commit-time 123 nonsense': 8000, + }); + }); + + it('ignores files listed before any commit header', () => { + expect(parseGitLog('stray.mdx\n', '', new Set()).size).toBe(0); + }); +}); + +describe('getSitemapEntries', () => { + it('covers every sitemap route with a valid date, or none at all', () => { + const entries = getSitemapEntries(); + expect(entries.length).toBeGreaterThan(100); + + const invalid = entries.filter((e) => e.lastModified !== undefined && Number.isNaN(e.lastModified.getTime())); + expect(invalid.map((e) => e.route)).toEqual([]); + + const unresolved = entries.filter((e) => !e.lastModified); + if (hasGitHistory()) { + // A full checkout can date everything; anything missing is a mapping bug. + expect(unresolved.map((e) => e.route)).toEqual([]); + } else { + // Degraded environment (no git, or shallow): blog posts still date + // themselves from frontmatter, and nothing is fabricated. + expect(entries.filter((e) => e.lastModified).every((e) => e.route.startsWith('/blog/'))).toBe(true); + } + }); + + it('dates a blog route no earlier than its publish date', () => { + const entry = getSitemapEntries().find((e) => e.route === '/blog/angular-chat-app-tutorial-with-ag-ui'); + expect(entry?.lastModified).toBeInstanceOf(Date); + // lastmod is last *modified*: the post's own file may have been edited after + // it was published, never before. + expect(entry?.lastModified?.getTime()).toBeGreaterThanOrEqual(Date.parse('2026-08-13T00:00:00Z')); + }); + + it('never claims a route changed in the future', () => { + const now = Date.now(); + const future = getSitemapEntries().filter((e) => (e.lastModified?.getTime() ?? 0) > now); + expect(future.map((e) => e.route)).toEqual([]); + }); + + it('resolves the special docs pages whose route shape differs from library docs', () => { + const entry = getSitemapEntries().find((e) => e.route === '/docs/choosing-an-adapter'); + if (hasGitHistory()) expect(entry?.lastModified).toBeInstanceOf(Date); + }); +}); + +describe('sitemap route', () => { + it('emits lastModified only — Google ignores changefreq and priority', () => { + const urls = sitemap(); + expect(urls.length).toBeGreaterThan(100); + + for (const url of urls) { + expect(url.url.startsWith('https://')).toBe(true); + expect('changeFrequency' in url).toBe(false); + expect('priority' in url).toBe(false); + } + + if (hasGitHistory()) { + expect(urls.every((url) => url.lastModified instanceof Date)).toBe(true); + } + }); +}); diff --git a/apps/website/src/lib/sitemap-dates.ts b/apps/website/src/lib/sitemap-dates.ts new file mode 100644 index 000000000..82045ae5a --- /dev/null +++ b/apps/website/src/lib/sitemap-dates.ts @@ -0,0 +1,242 @@ +// SPDX-License-Identifier: MIT +import { execFileSync } from 'node:child_process'; +import fs from 'node:fs'; +import path from 'node:path'; +import { getAllPosts, type Post } from './blog'; +import { specialDocsPages } from './docs-config'; +import { getSitemapRoutes } from './site-metadata'; +import { resolveWebsiteDir } from './website-dir'; + +/** A sitemap URL plus the honest last-modified time of the source it renders from. */ +export interface SitemapEntry { + route: string; + /** + * Omitted when no honest value can be determined. An absent `` is + * valid sitemap XML; a fabricated one (e.g. "now" on every build, which is + * what file mtimes give you on any fresh checkout) teaches crawlers to ignore + * the signal across the whole site. + */ + lastModified?: Date; +} + +/** Website-relative source paths that a route's content is rendered from. */ +function sourcePathsForRoute(route: string, postsBySlug: Map): string[] { + const post = postsBySlug.get(route); + if (post) return [path.join('content', 'blog', post.filename)]; + + const special = specialDocsPages.find((page) => page.path === route); + if (special) return [path.join('content', 'docs', special.contentPath)]; + + if (route.startsWith('/docs/')) { + const [, , library, section, slug] = route.split('/'); + if (library && section && slug) { + return [path.join('content', 'docs', library, section, `${slug}.mdx`)]; + } + // Anything else under /docs is a hand-written route, dated like any other. + } + + if (route.startsWith('/solutions/')) { + // Programmatic pages: one dynamic route template rendering data from a + // single module, so a change to either re-renders the page. + return [ + path.join('src', 'app', 'solutions', '[slug]', 'page.tsx'), + path.join('src', 'lib', 'solutions-data.ts'), + ]; + } + + const routeDir = route === '/' ? '' : route.replace(/^\//, ''); + return [path.join('src', 'app', routeDir, 'page.tsx')]; +} + +/** Absolute path for a website-relative source path, or null when it is missing. */ +function resolveSourcePath(relativePath: string): string | null { + const candidate = path.join(resolveWebsiteDir(), relativePath); + return fs.existsSync(candidate) ? candidate : null; +} + +const GIT_LOG_MARKER = 'commit-time '; +const GIT_LOG_HEADER = /^(\d+) ([0-9a-f]{40})$/; +const GIT_TIMEOUT_MS = 30_000; + +export interface GitTimes { + /** Last commit time (epoch seconds) keyed by website-relative path. */ + times: Map; + /** + * False when the clone is shallow. It says nothing about whether a given file + * appears in `times` — an absent path may be uncommitted, or may simply + * predate the visible history — which is why nothing here ever falls back to + * a file mtime. It only sharpens the build-log diagnostic. + */ + hasFullHistory: boolean; +} + +/** + * Parse `git log --format='commit-time %ct %H' --name-only` output into last + * commit time (epoch seconds) per repo path, with `prefix` stripped. + * + * Files attributed to a grafted (shallow-boundary) commit are dropped: such a + * commit has no parent, so git reports the *entire tree* as changed at its + * timestamp. Emitting those would claim the whole site changed at clone time, + * which is exactly the fabrication this module exists to avoid. + */ +export function parseGitLog(text: string, prefix: string, graftedShas: ReadonlySet): Map { + const times = new Map(); + let commitTime = 0; + + for (const line of text.split('\n')) { + if (line.startsWith(GIT_LOG_MARKER)) { + const header = GIT_LOG_HEADER.exec(line.slice(GIT_LOG_MARKER.length)); + // A path can legitimately begin with the marker text; only a well-formed + // " " remainder is a commit header. + if (header) { + commitTime = graftedShas.has(header[2]) ? 0 : Number(header[1]); + continue; + } + } + // Paths are emitted verbatim (core.quotePath=false), so leading whitespace + // is part of the name; only truly empty lines are separators. + if (line.length === 0 || !commitTime) continue; + const relative = prefix && line.startsWith(prefix) ? line.slice(prefix.length) : line; + // `git log` walks newest-first, so the first mention of a path wins. + if (!times.has(relative)) times.set(relative, commitTime); + } + + return times; +} + +/** + * Read last-commit times for the website's content and sources from git. + * + * Returns null when git history yields nothing usable (no git, not a repo, a + * clone so shallow that only grafted commits are visible, or a git invocation + * that overflows its buffer or times out — all of which throw rather than + * returning partial output). + */ +function computeGitTimes(): GitTimes | null { + try { + const git = (args: string[], cwd: string): string => + execFileSync( + 'git', + [ + // Emit paths verbatim instead of octal-escaping non-ASCII and quoting + // them, which would make every such path miss its lookup. + '-c', + 'core.quotePath=false', + // A user's `log.showSignature=true` would interleave `gpg:` lines + // into the file list. + '-c', + 'log.showSignature=false', + ...args, + ], + { + cwd, + encoding: 'utf8', + maxBuffer: 64 * 1024 * 1024, + timeout: GIT_TIMEOUT_MS, + killSignal: 'SIGKILL', + stdio: ['ignore', 'pipe', 'ignore'], + }, + ); + + const root = git(['rev-parse', '--show-toplevel'], process.cwd()).trim(); + const graftedShas = new Set(); + const shallow = git(['rev-parse', '--is-shallow-repository'], root).trim() === 'true'; + if (shallow) { + // Must be the COMMON git dir, not `--absolute-git-dir`: in a linked + // worktree those differ (`.git/worktrees/` vs `.git`) and `shallow` + // only ever lives in the common dir. Output can be relative, and git runs + // with cwd=root, so resolve it against root. + const commonDir = path.resolve(root, git(['rev-parse', '--git-common-dir'], root).trim()); + const shallowFile = path.join(commonDir, 'shallow'); + if (!fs.existsSync(shallowFile)) return null; + for (const sha of fs.readFileSync(shallowFile, 'utf8').split('\n')) { + if (sha.trim()) graftedShas.add(sha.trim()); + } + } + + const prefix = fs.existsSync(path.join(root, 'apps', 'website')) ? 'apps/website/' : ''; + const log = git( + [ + 'log', + `--format=${GIT_LOG_MARKER}%ct %H`, + '--name-only', + '--', + `${prefix}content`, + `${prefix}src/app`, + `${prefix}src/lib`, + ], + root, + ); + + const times = parseGitLog(log, prefix, graftedShas); + return times.size > 0 ? { times, hasFullHistory: !shallow } : null; + } catch { + return null; + } +} + +let gitTimesCache: GitTimes | null | undefined; + +function getGitTimes(): GitTimes | null { + if (gitTimesCache !== undefined) return gitTimesCache; + gitTimesCache = computeGitTimes(); + + if (gitTimesCache === null) { + // Visible in the build log: the sitemap will carry `lastmod` only for blog + // posts (which date themselves in frontmatter) until history is available. + console.warn('[sitemap] git history unavailable; omitting for file-derived routes.'); + } else if (!gitTimesCache.hasFullHistory) { + console.warn('[sitemap] shallow clone; omitting for routes older than the visible history.'); + } + + return gitTimesCache; +} + +/** True when git can date at least some sources. Exposed for tests. */ +export function hasGitHistory(): boolean { + return getGitTimes() !== null; +} + +/** + * Newest commit time across a route's sources, or undefined when git cannot + * date any of them. File mtimes are deliberately never consulted: any fresh + * clone — shallow or not — rewrites every mtime to checkout time, so a single + * lookup miss would silently publish a build-time `lastmod`. + */ +function sourceModifiedTime(relativePaths: string[]): Date | undefined { + const git = getGitTimes(); + if (!git) return undefined; + + let newest: number | undefined; + for (const relativePath of relativePaths) { + if (!resolveSourcePath(relativePath)) continue; + const seconds = git.times.get(relativePath.split(path.sep).join('/')); + if (seconds === undefined) continue; + if (newest === undefined || seconds > newest) newest = seconds; + } + + return newest === undefined ? undefined : new Date(newest * 1000); +} + +/** Frontmatter publish date as UTC midnight, or undefined when unparseable. */ +function publishedDate(post: Post): Date | undefined { + const date = new Date(`${post.frontmatter.date}T00:00:00Z`); + return Number.isNaN(date.getTime()) ? undefined : date; +} + +export function getSitemapEntries(): SitemapEntry[] { + const postsBySlug = new Map(getAllPosts().map((post) => [`/blog/${post.slug}`, post])); + + return getSitemapRoutes().map((route) => { + const committed = sourceModifiedTime(sourcePathsForRoute(route, postsBySlug)); + + // `lastmod` means last *modified*, so an edited post outranks its own + // publish date; the frontmatter date still covers posts git cannot date. + const post = postsBySlug.get(route); + const published = post ? publishedDate(post) : undefined; + const lastModified = + committed && published ? (committed > published ? committed : published) : (committed ?? published); + + return lastModified ? { route, lastModified } : { route }; + }); +} diff --git a/apps/website/src/lib/website-dir.ts b/apps/website/src/lib/website-dir.ts new file mode 100644 index 000000000..605184dbc --- /dev/null +++ b/apps/website/src/lib/website-dir.ts @@ -0,0 +1,12 @@ +// SPDX-License-Identifier: MIT +import fs from 'node:fs'; +import path from 'node:path'; + +/** + * Absolute path to `apps/website`, whether the process was started from the + * workspace root (`nx build website`) or from the app itself (`next build`). + */ +export function resolveWebsiteDir(): string { + const workspace = path.join(process.cwd(), 'apps', 'website'); + return fs.existsSync(workspace) ? workspace : process.cwd(); +} From 532b8b17a32fbb06b80451f9537e23463eb18048 Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 20 Aug 2026 12:45:43 -0700 Subject: [PATCH 14/35] feat(website): article metadata + canonical brand spelling Emit article:published_time / modified_time / author / tag on blog posts, with modified_time derived from real git commit times (sitemap-dates) rather than defaulting to the publish date. createPageMetadata gains an optional per-page social image for the Task 9 OG-image routes. Unify the brand on "Threadplane" (was "ThreadPlane" in blog titles and prose) and the docs title separator on an em dash. Co-Authored-By: Claude Opus 5 --- ...stack-agentic-angular-apps-using-ag-ui.mdx | 6 +-- apps/website/e2e/blog.spec.ts | 2 +- apps/website/e2e/website.spec.ts | 4 +- apps/website/src/app/blog/[slug]/page.tsx | 24 +++++++++-- apps/website/src/app/blog/page.tsx | 4 +- .../docs/[library]/[section]/[slug]/page.tsx | 2 +- apps/website/src/lib/docs.spec.ts | 6 +-- apps/website/src/lib/docs.ts | 2 +- apps/website/src/lib/site-metadata.spec.ts | 40 +++++++++++++++++++ apps/website/src/lib/site-metadata.ts | 29 +++++++++++++- apps/website/src/lib/sitemap-dates.ts | 31 +++++++++----- 11 files changed, 123 insertions(+), 27 deletions(-) diff --git a/apps/website/content/blog/2026-05-21-build-fullstack-agentic-angular-apps-using-ag-ui.mdx b/apps/website/content/blog/2026-05-21-build-fullstack-agentic-angular-apps-using-ag-ui.mdx index 87be3b491..a790bf58f 100644 --- a/apps/website/content/blog/2026-05-21-build-fullstack-agentic-angular-apps-using-ag-ui.mdx +++ b/apps/website/content/blog/2026-05-21-build-fullstack-agentic-angular-apps-using-ag-ui.mdx @@ -57,7 +57,7 @@ Three boxes. Two seams. **The wire.** Server-Sent Events. Plain HTTP, no WebSocket gymnastics, no custom binary framing. Your firewall, load balancer, and reverse proxy already know what to do with it. -**The Angular side.** This is what ThreadPlane provides. `@threadplane/ag-ui` is the adapter. It consumes the AG-UI event stream and exposes a runtime-neutral `Agent` contract built from signals. `@threadplane/chat` is the UI. It reads from that contract and renders. The two are decoupled on purpose. We'll get to why. +**The Angular side.** This is what Threadplane provides. `@threadplane/ag-ui` is the adapter. It consumes the AG-UI event stream and exposes a runtime-neutral `Agent` contract built from signals. `@threadplane/chat` is the UI. It reads from that contract and renders. The two are decoupled on purpose. We'll get to why. ## Let's wire it up @@ -164,7 +164,7 @@ The AG-UI protocol has seventeen event types, grouped into five families: The families each do specific work. Lifecycle answers "is something happening?" Text messages are the streaming triad familiar from chat UIs. Tool calls are deliberately incremental so you can render the *intent* before the arguments are fully formed. State sync uses RFC 6902 JSON Patch so the wire stays small even when the agent's state is large. -ThreadPlane's `@threadplane/ag-ui` runs each event through a small reducer that updates a handful of signals on the `Agent` contract: +Threadplane's `@threadplane/ag-ui` runs each event through a small reducer that updates a handful of signals on the `Agent` contract: - `messages()`: `Message[]`, the chat history. `TEXT_MESSAGE_CONTENT` appends a delta to the in-flight assistant message. - `status()`: `'idle' | 'running' | 'error' | 'paused'`. Driven by the `RUN_*` events. @@ -300,6 +300,6 @@ Each of those is its own post. The point here is just that the protocol-to-signa ## Conclusion -AG-UI standardizes the wire between the agent and the UI: it's small enough to hold in your head, and the event model maps onto Angular signals cleanly. With ThreadPlane (`@threadplane/ag-ui` and `@threadplane/chat` on npm), the wiring is three lines — a provider, an inject, and a `` — which leaves the interesting work (tool cards, interrupt flows, generative UI, your design system) as the part you spend the day on. +AG-UI standardizes the wire between the agent and the UI: it's small enough to hold in your head, and the event model maps onto Angular signals cleanly. With Threadplane (`@threadplane/ag-ui` and `@threadplane/chat` on npm), the wiring is three lines — a provider, an inject, and a `` — which leaves the interesting work (tool cards, interrupt flows, generative UI, your design system) as the part you spend the day on. The adapters are MIT; `@threadplane/chat` is source-available with a free non-commercial tier. If you're building this inside an enterprise Angular app (design system, multi-tenant, regulated), [talk to us](/contact?source=blog_ag_ui_pillar&track=enterprise). diff --git a/apps/website/e2e/blog.spec.ts b/apps/website/e2e/blog.spec.ts index 1c187dda9..2b667d485 100644 --- a/apps/website/e2e/blog.spec.ts +++ b/apps/website/e2e/blog.spec.ts @@ -6,7 +6,7 @@ test.describe('Blog landing page', () => { // Brand eyebrow + H1 await expect(page.getByText('Blog', { exact: true }).first()).toBeVisible(); - await expect(page.getByRole('heading', { level: 1, name: /Articles from ThreadPlane/i })).toBeVisible(); + await expect(page.getByRole('heading', { level: 1, name: /Articles from Threadplane/i })).toBeVisible(); // Filter row contains the "All" chip in active state await expect(page.getByText('All', { exact: true })).toBeVisible(); diff --git a/apps/website/e2e/website.spec.ts b/apps/website/e2e/website.spec.ts index 21490729f..038aa67a4 100644 --- a/apps/website/e2e/website.spec.ts +++ b/apps/website/e2e/website.spec.ts @@ -237,7 +237,7 @@ test('docs pages render canonical and social metadata', async ({ page }) => { ); await expect(page.locator('meta[property="og:title"]')).toHaveAttribute( 'content', - 'Streaming - LangGraph Docs - Threadplane', + 'Streaming — LangGraph Docs — Threadplane', ); await expect(page.locator('meta[property="og:url"]')).toHaveAttribute( 'content', @@ -245,7 +245,7 @@ test('docs pages render canonical and social metadata', async ({ page }) => { ); await expect(page.locator('meta[name="twitter:title"]')).toHaveAttribute( 'content', - 'Streaming - LangGraph Docs - Threadplane', + 'Streaming — LangGraph Docs — Threadplane', ); }); diff --git a/apps/website/src/app/blog/[slug]/page.tsx b/apps/website/src/app/blog/[slug]/page.tsx index d666fd023..408b01332 100644 --- a/apps/website/src/app/blog/[slug]/page.tsx +++ b/apps/website/src/app/blog/[slug]/page.tsx @@ -10,6 +10,7 @@ import { getAllPosts, getPostBySlug, formatPostDate, readingTimeMin } from '../. import { getAuthor } from '../../../lib/blog-authors'; import { extractHeadings } from '../../../lib/extract-headings'; import { createPageMetadata } from '../../../lib/site-metadata'; +import { getRouteLastModified } from '../../../lib/sitemap-dates'; interface Params { params: Promise<{ slug: string }>; @@ -23,13 +24,30 @@ export async function generateMetadata({ params }: Params): Promise { const { slug } = await params; const post = getPostBySlug(slug); if (!post || post.frontmatter.draft) { - return { title: 'Post not found — ThreadPlane' }; + return { title: 'Post not found — Threadplane' }; } + const author = getAuthor(post.frontmatter.author); + const pathname = `/blog/${post.slug}`; + // Only claim a modification when git actually shows the post was edited after + // its publish date; otherwise `createPageMetadata` reports it unmodified. + const published = new Date(`${post.frontmatter.date}T00:00:00Z`); + const lastModified = getRouteLastModified(pathname); + const modifiedTime = + lastModified && lastModified.getTime() > published.getTime() ? lastModified.toISOString() : undefined; + return createPageMetadata({ - title: `${post.frontmatter.title} — ThreadPlane`, + title: `${post.frontmatter.title} — Threadplane`, description: post.frontmatter.description, - pathname: `/blog/${post.slug}`, + pathname, type: 'article', + // TODO(task 9): point at `${pathname}/opengraph-image` once that route + // exists; naming it before then would emit an og:image URL that 404s. + article: { + publishedTime: post.frontmatter.date, + modifiedTime, + authors: [author.name], + tags: post.frontmatter.tags, + }, }); } diff --git a/apps/website/src/app/blog/page.tsx b/apps/website/src/app/blog/page.tsx index 212e09916..67a2bb25d 100644 --- a/apps/website/src/app/blog/page.tsx +++ b/apps/website/src/app/blog/page.tsx @@ -8,7 +8,7 @@ import { BlogTagFilter } from '../../components/blog/BlogTagFilter'; import { Eyebrow } from '../../components/ui/Eyebrow'; export const metadata = createPageMetadata({ - title: 'Blog — ThreadPlane', + title: 'Blog — Threadplane', description: 'Long-form writing on agent UI for Angular: streaming, generative UI, threads, interrupts, production patterns.', pathname: '/blog', @@ -51,7 +51,7 @@ export default async function BlogIndexPage({ searchParams }: Props) { margin: '0 0 16px', }} > - Articles from ThreadPlane + Articles from Threadplane

{ const { library, section, slug } = await params; return getDocMetadata(library, section, slug) ?? { - title: 'Docs - Threadplane', + title: 'Docs — Threadplane', description: 'Threadplane documentation', }; } diff --git a/apps/website/src/lib/docs.spec.ts b/apps/website/src/lib/docs.spec.ts index 9616730ab..2a986ecd2 100644 --- a/apps/website/src/lib/docs.spec.ts +++ b/apps/website/src/lib/docs.spec.ts @@ -99,17 +99,17 @@ describe('website docs bindings', () => { const metadata = getDocMetadata('ag-ui', 'reference', 'event-mapping'); expect(metadata).toMatchObject({ - title: 'Event Mapping - AG-UI Docs - Threadplane', + title: 'Event Mapping — AG-UI Docs — Threadplane', alternates: { canonical: '/docs/ag-ui/reference/event-mapping', }, openGraph: { - title: 'Event Mapping - AG-UI Docs - Threadplane', + title: 'Event Mapping — AG-UI Docs — Threadplane', url: '/docs/ag-ui/reference/event-mapping', }, twitter: { card: 'summary_large_image', - title: 'Event Mapping - AG-UI Docs - Threadplane', + title: 'Event Mapping — AG-UI Docs — Threadplane', }, }); expect(metadata?.description).toContain('AG-UI protocol events'); diff --git a/apps/website/src/lib/docs.ts b/apps/website/src/lib/docs.ts index 28d637541..02b57ec13 100644 --- a/apps/website/src/lib/docs.ts +++ b/apps/website/src/lib/docs.ts @@ -86,7 +86,7 @@ export function getDocMetadata( const lib = getLibraryConfig(library); const libraryTitle = lib?.title ?? 'Docs'; - const title = `${doc.title} - ${libraryTitle} Docs - Threadplane`; + const title = `${doc.title} — ${libraryTitle} Docs — Threadplane`; const description = getDocDescription(doc.content, lib?.description ?? 'Threadplane documentation'); const pathname = `/docs/${library}/${section}/${slug}`; diff --git a/apps/website/src/lib/site-metadata.spec.ts b/apps/website/src/lib/site-metadata.spec.ts index 5e9db745d..7984383fa 100644 --- a/apps/website/src/lib/site-metadata.spec.ts +++ b/apps/website/src/lib/site-metadata.spec.ts @@ -6,6 +6,7 @@ import { POSITIONING_PROOF_POINTS, PRIMARY_TAGLINE, SHORT_POSITIONING_DESCRIPTION, + SITE_NAME, createPageMetadata, } from './site-metadata'; @@ -49,3 +50,42 @@ describe('site positioning copy', () => { expect(metadata.twitter?.description).toBe(DEFAULT_META_DESCRIPTION); }); }); + +describe('createPageMetadata article fields', () => { + it('emits openGraph article dates, authors, and tags', () => { + const metadata = createPageMetadata({ + title: 'Post — Threadplane', + description: 'A post.', + pathname: '/blog/post', + type: 'article', + article: { + publishedTime: '2026-08-13', + modifiedTime: '2026-08-14', + authors: ['Brian Love'], + tags: ['angular', 'ag-ui'], + }, + }); + const openGraph = metadata.openGraph as Record; + expect(openGraph['publishedTime']).toBe('2026-08-13'); + expect(openGraph['modifiedTime']).toBe('2026-08-14'); + expect(openGraph['authors']).toEqual(['Brian Love']); + expect(openGraph['tags']).toEqual(['angular', 'ag-ui']); + }); + + it('accepts a page-specific social image', () => { + const metadata = createPageMetadata({ + title: 'Post — Threadplane', + description: 'A post.', + pathname: '/blog/post', + image: '/blog/post/opengraph-image', + }); + const openGraph = metadata.openGraph as { images: string[] }; + expect(openGraph.images).toEqual(['/blog/post/opengraph-image']); + }); +}); + +describe('brand name', () => { + it('uses one canonical spelling', () => { + expect(SITE_NAME).toBe('Threadplane'); + }); +}); diff --git a/apps/website/src/lib/site-metadata.ts b/apps/website/src/lib/site-metadata.ts index 6a5744005..ce9260e11 100644 --- a/apps/website/src/lib/site-metadata.ts +++ b/apps/website/src/lib/site-metadata.ts @@ -25,16 +25,33 @@ export function getCanonicalUrl(pathname: string): string { return new URL(getCanonicalPath(pathname), SITE_ORIGIN).toString(); } +/** Article-specific OpenGraph fields (freshness + attribution signals). */ +export interface ArticleMetadata { + /** ISO 8601 publish date or timestamp. */ + publishedTime: string; + /** + * ISO 8601 last-modified timestamp. Omit only when the content genuinely has + * not changed since publication — it then falls back to `publishedTime`. + */ + modifiedTime?: string; + authors?: string[]; + tags?: string[]; +} + export function createPageMetadata({ title, description, pathname, type = 'article', + image = DEFAULT_SOCIAL_IMAGE, + article, }: { title: string; description: string; pathname: string; type?: 'article' | 'website'; + image?: string; + article?: ArticleMetadata; }): Metadata { const canonicalPath = getCanonicalPath(pathname); @@ -50,13 +67,21 @@ export function createPageMetadata({ url: canonicalPath, siteName: SITE_NAME, type, - images: [DEFAULT_SOCIAL_IMAGE], + images: [image], + ...(article + ? { + publishedTime: article.publishedTime, + modifiedTime: article.modifiedTime ?? article.publishedTime, + authors: article.authors, + tags: article.tags, + } + : {}), }, twitter: { card: 'summary_large_image', title, description, - images: [DEFAULT_SOCIAL_IMAGE], + images: [image], }, }; } diff --git a/apps/website/src/lib/sitemap-dates.ts b/apps/website/src/lib/sitemap-dates.ts index 82045ae5a..cc4f7d283 100644 --- a/apps/website/src/lib/sitemap-dates.ts +++ b/apps/website/src/lib/sitemap-dates.ts @@ -224,19 +224,32 @@ function publishedDate(post: Post): Date | undefined { return Number.isNaN(date.getTime()) ? undefined : date; } +/** + * Honest last-modified time for one route, or undefined when nothing can date + * it. Shared by the sitemap and by article metadata (`og:modified_time`), so + * both surfaces make the same freshness claim. + * + * `postsBySlug` is an optional cache: callers walking every route build it once + * rather than re-reading the blog directory per route. + */ +export function getRouteLastModified(route: string, postsBySlug?: Map): Date | undefined { + const posts = + postsBySlug ?? + new Map(getAllPosts({ includeDrafts: true }).map((post) => [`/blog/${post.slug}`, post])); + const committed = sourceModifiedTime(sourcePathsForRoute(route, posts)); + + // "Modified" means last *modified*, so an edited post outranks its own + // publish date; the frontmatter date still covers posts git cannot date. + const post = posts.get(route); + const published = post ? publishedDate(post) : undefined; + return committed && published ? (committed > published ? committed : published) : (committed ?? published); +} + export function getSitemapEntries(): SitemapEntry[] { const postsBySlug = new Map(getAllPosts().map((post) => [`/blog/${post.slug}`, post])); return getSitemapRoutes().map((route) => { - const committed = sourceModifiedTime(sourcePathsForRoute(route, postsBySlug)); - - // `lastmod` means last *modified*, so an edited post outranks its own - // publish date; the frontmatter date still covers posts git cannot date. - const post = postsBySlug.get(route); - const published = post ? publishedDate(post) : undefined; - const lastModified = - committed && published ? (committed > published ? committed : published) : (committed ?? published); - + const lastModified = getRouteLastModified(route, postsBySlug); return lastModified ? { route, lastModified } : { route }; }); } From 7c88eaeb6fb2c6b731a275eea011229ce3dd1eb0 Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 20 Aug 2026 12:55:17 -0700 Subject: [PATCH 15/35] refactor(website): single-source the blog modified time Drop the redundant `lastModified > published` guard in the blog route: it could never change the value (getRouteLastModified already returns the max) and its only effect was flipping modifiedTime to undefined, which made unedited posts emit a date-only article:published_time while edited ones emitted a full ISO timestamp. The `?? publishedTime` fallback in createPageMetadata is now the one place that rule lives, and both fields are ISO timestamps on every post. Date parsing moves to the shared `publishedDate()` helper, so a malformed frontmatter date drops the article block instead of shipping NaN-adjacent garbage as article:published_time. getRouteLastModified's post index is now a required `postsByRoute` (renamed: the keys are route paths) rather than an optional argument with a drafts policy that diverged from the caller's, which could return different answers for the same route. Callers build it with the new getPostsByRoute() or pass the post they already hold. Replaces the tautological SITE_NAME assertion with a real scan of src, content, scripts, and e2e for the mis-cased brand, and covers the modifiedTime fallback plus the absence of article keys on non-article pages. Co-Authored-By: Claude Opus 5 --- apps/website/src/app/blog/[slug]/page.tsx | 30 ++++----- apps/website/src/lib/site-metadata.spec.ts | 72 +++++++++++++++++++++- apps/website/src/lib/site-metadata.ts | 41 ++++++------ apps/website/src/lib/sitemap-dates.ts | 37 ++++++----- 4 files changed, 132 insertions(+), 48 deletions(-) diff --git a/apps/website/src/app/blog/[slug]/page.tsx b/apps/website/src/app/blog/[slug]/page.tsx index 408b01332..a3bed5312 100644 --- a/apps/website/src/app/blog/[slug]/page.tsx +++ b/apps/website/src/app/blog/[slug]/page.tsx @@ -10,7 +10,7 @@ import { getAllPosts, getPostBySlug, formatPostDate, readingTimeMin } from '../. import { getAuthor } from '../../../lib/blog-authors'; import { extractHeadings } from '../../../lib/extract-headings'; import { createPageMetadata } from '../../../lib/site-metadata'; -import { getRouteLastModified } from '../../../lib/sitemap-dates'; +import { getRouteLastModified, publishedDate } from '../../../lib/sitemap-dates'; interface Params { params: Promise<{ slug: string }>; @@ -28,26 +28,28 @@ export async function generateMetadata({ params }: Params): Promise { } const author = getAuthor(post.frontmatter.author); const pathname = `/blog/${post.slug}`; - // Only claim a modification when git actually shows the post was edited after - // its publish date; otherwise `createPageMetadata` reports it unmodified. - const published = new Date(`${post.frontmatter.date}T00:00:00Z`); - const lastModified = getRouteLastModified(pathname); - const modifiedTime = - lastModified && lastModified.getTime() > published.getTime() ? lastModified.toISOString() : undefined; + // The post is already resolved, so index just it: no drafts policy to inherit + // and no second read of the blog directory. + const lastModified = getRouteLastModified(pathname, new Map([[pathname, post]])); + // Undefined for an unparseable frontmatter date, which drops the article + // block rather than shipping the bad string as `article:published_time`. + const published = publishedDate(post); return createPageMetadata({ title: `${post.frontmatter.title} — Threadplane`, description: post.frontmatter.description, pathname, type: 'article', - // TODO(task 9): point at `${pathname}/opengraph-image` once that route + // TODO(task 9): set image to "/opengraph-image" once that route // exists; naming it before then would emit an og:image URL that 404s. - article: { - publishedTime: post.frontmatter.date, - modifiedTime, - authors: [author.name], - tags: post.frontmatter.tags, - }, + article: published + ? { + publishedTime: published.toISOString(), + modifiedTime: lastModified?.toISOString(), + authors: [author.name], + tags: post.frontmatter.tags, + } + : undefined, }); } diff --git a/apps/website/src/lib/site-metadata.spec.ts b/apps/website/src/lib/site-metadata.spec.ts index 7984383fa..84559a11c 100644 --- a/apps/website/src/lib/site-metadata.spec.ts +++ b/apps/website/src/lib/site-metadata.spec.ts @@ -1,3 +1,5 @@ +import fs from 'node:fs'; +import path from 'node:path'; import { describe, expect, it } from 'vitest'; import { DEFAULT_META_DESCRIPTION, @@ -9,6 +11,7 @@ import { SITE_NAME, createPageMetadata, } from './site-metadata'; +import { resolveWebsiteDir } from './website-dir'; describe('site positioning copy', () => { it('exports the approved primary tagline and supporting copy', () => { @@ -84,8 +87,75 @@ describe('createPageMetadata article fields', () => { }); }); +describe('createPageMetadata article fallback', () => { + it('advertises the publish date as the modification when none is known', () => { + const metadata = createPageMetadata({ + title: 'Post — Threadplane', + description: 'A post.', + pathname: '/blog/post', + article: { publishedTime: '2026-08-13T00:00:00.000Z' }, + }); + const openGraph = metadata.openGraph as Record; + expect(openGraph['modifiedTime']).toBe('2026-08-13T00:00:00.000Z'); + }); + + it('omits article fields entirely for a non-article page', () => { + const metadata = createPageMetadata({ + title: 'Home — Threadplane', + description: 'A page.', + pathname: '/', + type: 'website', + }); + const openGraph = metadata.openGraph as Record; + // Absent, not undefined: Next emits a meta tag for a present-but-undefined + // key in some shapes, and a landing page has no publication to date. + expect('publishedTime' in openGraph).toBe(false); + expect('modifiedTime' in openGraph).toBe(false); + expect('authors' in openGraph).toBe(false); + expect('tags' in openGraph).toBe(false); + }); +}); + describe('brand name', () => { - it('uses one canonical spelling', () => { + // Built at runtime so this spec file — which lives under a scanned root — + // does not match its own needle. + const MISSPELLING = new RegExp(['Thread', 'Plane'].join(''), 'g'); + const SCAN_ROOTS = ['src', 'content', 'scripts', 'e2e']; + const ALLOWED = new Set([ + // A deliberate mixed-case URL fixture for host-case normalization. + path.join('scripts', 'gsc', 'analysis.spec.ts'), + ]); + const SKIP_DIRS = new Set(['node_modules', '.next', 'dist']); + + function walk(dir: string, root: string, out: string[]): string[] { + for (const entry of fs.readdirSync(dir, { withFileTypes: true })) { + const full = path.join(dir, entry.name); + if (entry.isDirectory()) { + if (!SKIP_DIRS.has(entry.name)) walk(full, root, out); + } else if (entry.isFile()) { + out.push(path.relative(root, full)); + } + } + return out; + } + + it('is spelled the same way everywhere it ships', () => { + const websiteDir = resolveWebsiteDir(); + const offenders: string[] = []; + + for (const root of SCAN_ROOTS) { + const rootDir = path.join(websiteDir, root); + if (!fs.existsSync(rootDir)) continue; + for (const relative of walk(rootDir, websiteDir, [])) { + if (ALLOWED.has(relative)) continue; + if (MISSPELLING.test(fs.readFileSync(path.join(websiteDir, relative), 'utf8'))) { + offenders.push(relative); + } + MISSPELLING.lastIndex = 0; + } + } + + expect(offenders).toEqual([]); expect(SITE_NAME).toBe('Threadplane'); }); }); diff --git a/apps/website/src/lib/site-metadata.ts b/apps/website/src/lib/site-metadata.ts index ce9260e11..28e7809cd 100644 --- a/apps/website/src/lib/site-metadata.ts +++ b/apps/website/src/lib/site-metadata.ts @@ -30,14 +30,26 @@ export interface ArticleMetadata { /** ISO 8601 publish date or timestamp. */ publishedTime: string; /** - * ISO 8601 last-modified timestamp. Omit only when the content genuinely has - * not changed since publication — it then falls back to `publishedTime`. + * ISO 8601 last-modified timestamp. Omit when no modification is known; it + * then falls back to `publishedTime`. */ modifiedTime?: string; authors?: string[]; tags?: string[]; } +/** Options for {@link createPageMetadata}. */ +export interface PageMetadataOptions { + title: string; + description: string; + pathname: string; + type?: 'article' | 'website'; + /** Social image path; resolved against `metadataBase` from the root layout. */ + image?: string; + /** Present only for article-type pages; omitted entirely for landing pages. */ + article?: ArticleMetadata; +} + export function createPageMetadata({ title, description, @@ -45,14 +57,7 @@ export function createPageMetadata({ type = 'article', image = DEFAULT_SOCIAL_IMAGE, article, -}: { - title: string; - description: string; - pathname: string; - type?: 'article' | 'website'; - image?: string; - article?: ArticleMetadata; -}): Metadata { +}: PageMetadataOptions): Metadata { const canonicalPath = getCanonicalPath(pathname); return { @@ -68,14 +73,14 @@ export function createPageMetadata({ siteName: SITE_NAME, type, images: [image], - ...(article - ? { - publishedTime: article.publishedTime, - modifiedTime: article.modifiedTime ?? article.publishedTime, - authors: article.authors, - tags: article.tags, - } - : {}), + ...(article && { + publishedTime: article.publishedTime, + // The single place the "unmodified" rule lives: an article with no + // known modification advertises its publish date as the modification. + modifiedTime: article.modifiedTime ?? article.publishedTime, + authors: article.authors, + tags: article.tags, + }), }, twitter: { card: 'summary_large_image', diff --git a/apps/website/src/lib/sitemap-dates.ts b/apps/website/src/lib/sitemap-dates.ts index cc4f7d283..9b4fd502b 100644 --- a/apps/website/src/lib/sitemap-dates.ts +++ b/apps/website/src/lib/sitemap-dates.ts @@ -20,8 +20,8 @@ export interface SitemapEntry { } /** Website-relative source paths that a route's content is rendered from. */ -function sourcePathsForRoute(route: string, postsBySlug: Map): string[] { - const post = postsBySlug.get(route); +function sourcePathsForRoute(route: string, postsByRoute: ReadonlyMap): string[] { + const post = postsByRoute.get(route); if (post) return [path.join('content', 'blog', post.filename)]; const special = specialDocsPages.find((page) => page.path === route); @@ -219,37 +219,44 @@ function sourceModifiedTime(relativePaths: string[]): Date | undefined { } /** Frontmatter publish date as UTC midnight, or undefined when unparseable. */ -function publishedDate(post: Post): Date | undefined { +export function publishedDate(post: Post): Date | undefined { const date = new Date(`${post.frontmatter.date}T00:00:00Z`); return Number.isNaN(date.getTime()) ? undefined : date; } +/** Index the posts a caller wants visible, keyed by route path. */ +export function getPostsByRoute(options: { includeDrafts?: boolean } = {}): Map { + return new Map(getAllPosts(options).map((post) => [`/blog/${post.slug}`, post])); +} + /** * Honest last-modified time for one route, or undefined when nothing can date - * it. Shared by the sitemap and by article metadata (`og:modified_time`), so - * both surfaces make the same freshness claim. + * it. Shared by the sitemap and by article metadata (`article:modified_time`), + * so both surfaces make the same freshness claim for the same route. * - * `postsBySlug` is an optional cache: callers walking every route build it once - * rather than re-reading the blog directory per route. + * `postsByRoute` is required rather than defaulted: a route's answer depends on + * whether its post is visible in the map (a missing one loses its frontmatter + * date and falls through to a page source that does not exist for blog posts), + * so every caller states which posts it means instead of inheriting a drafts + * policy chosen here. Build it with {@link getPostsByRoute}, or pass a + * single-entry map when the post is already in hand. */ -export function getRouteLastModified(route: string, postsBySlug?: Map): Date | undefined { - const posts = - postsBySlug ?? - new Map(getAllPosts({ includeDrafts: true }).map((post) => [`/blog/${post.slug}`, post])); - const committed = sourceModifiedTime(sourcePathsForRoute(route, posts)); +export function getRouteLastModified(route: string, postsByRoute: ReadonlyMap): Date | undefined { + const committed = sourceModifiedTime(sourcePathsForRoute(route, postsByRoute)); // "Modified" means last *modified*, so an edited post outranks its own // publish date; the frontmatter date still covers posts git cannot date. - const post = posts.get(route); + const post = postsByRoute.get(route); const published = post ? publishedDate(post) : undefined; return committed && published ? (committed > published ? committed : published) : (committed ?? published); } export function getSitemapEntries(): SitemapEntry[] { - const postsBySlug = new Map(getAllPosts().map((post) => [`/blog/${post.slug}`, post])); + // Drafts are not in `getSitemapRoutes()`, so they are not indexed either. + const postsByRoute = getPostsByRoute(); return getSitemapRoutes().map((route) => { - const lastModified = getRouteLastModified(route, postsBySlug); + const lastModified = getRouteLastModified(route, postsByRoute); return lastModified ? { route, lastModified } : { route }; }); } From 65aa7c7bf0f9c60b8fbb9a33dec4f70750f77fea Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 20 Aug 2026 13:00:41 -0700 Subject: [PATCH 16/35] feat(website): schema.org json-ld builders Pure builders for Organization, WebSite, SoftwareSourceCode, BlogPosting, TechArticle, and BreadcrumbList nodes, plus a JsonLd render component. Not mounted on any route yet. Every emitted URL was verified to resolve: the repository is cacheplane/angular-agent-framework (not blove/...), sameAs links the public package page rather than the member-gated npm org page, and the author URL and per-post OG image are omitted until /about and the fixed image route exist. Co-Authored-By: Claude Opus 5 --- .../src/components/shared/JsonLd.spec.tsx | 46 +++++ apps/website/src/components/shared/JsonLd.tsx | 13 ++ apps/website/src/lib/structured-data.spec.ts | 169 ++++++++++++++++++ apps/website/src/lib/structured-data.ts | 149 +++++++++++++++ 4 files changed, 377 insertions(+) create mode 100644 apps/website/src/components/shared/JsonLd.spec.tsx create mode 100644 apps/website/src/components/shared/JsonLd.tsx create mode 100644 apps/website/src/lib/structured-data.spec.ts create mode 100644 apps/website/src/lib/structured-data.ts diff --git a/apps/website/src/components/shared/JsonLd.spec.tsx b/apps/website/src/components/shared/JsonLd.spec.tsx new file mode 100644 index 000000000..b8626a6d2 --- /dev/null +++ b/apps/website/src/components/shared/JsonLd.spec.tsx @@ -0,0 +1,46 @@ +// @vitest-environment jsdom +// SPDX-License-Identifier: MIT +import { describe, expect, it } from 'vitest'; +import { render } from '@testing-library/react'; +import { JsonLd } from './JsonLd'; + +function renderScript(data: Parameters[0]['data']): HTMLScriptElement { + const { container } = render(); + const script = container.querySelector('script[type="application/ld+json"]'); + expect(script).not.toBeNull(); + return script as HTMLScriptElement; +} + +describe('JsonLd', () => { + it('emits a parseable ld+json script', () => { + const script = renderScript({ '@type': 'Organization', name: 'Threadplane' }); + expect(JSON.parse(script.textContent ?? '')).toEqual({ + '@type': 'Organization', + name: 'Threadplane', + }); + }); + + it('accepts an array of nodes', () => { + const script = renderScript([{ '@type': 'WebSite' }, { '@type': 'Organization' }]); + expect(JSON.parse(script.textContent ?? '')).toHaveLength(2); + }); + + it('escapes < so a value cannot close the script tag', () => { + const hostile = ''; + const script = renderScript({ '@type': 'Organization', name: hostile }); + + // The raw markup must contain no literal `<` inside the script body... + expect(script.innerHTML).not.toContain('<'); + expect(script.innerHTML).toContain('\\u003c/script>'); + // ...and the escape is lossless: the value round-trips exactly. + expect(JSON.parse(script.textContent ?? '')['name']).toBe(hostile); + }); + + it('does not inject a sibling element when a value looks like markup', () => { + const { container } = render( + x' }} />, + ); + expect(container.querySelector('#pwned')).toBeNull(); + expect(container.childElementCount).toBe(1); + }); +}); diff --git a/apps/website/src/components/shared/JsonLd.tsx b/apps/website/src/components/shared/JsonLd.tsx new file mode 100644 index 000000000..8b46e224a --- /dev/null +++ b/apps/website/src/components/shared/JsonLd.tsx @@ -0,0 +1,13 @@ +// SPDX-License-Identifier: MIT +import type { JsonLdNode } from '../../lib/structured-data'; + +/** + * Renders schema.org JSON-LD. Content is generated from our own data, never + * from user input, so `dangerouslySetInnerHTML` is safe here; `<` is still + * escaped so a stray value cannot close the script tag. `<` is a valid + * JSON string escape, so the escaped payload still parses back identically. + */ +export function JsonLd({ data }: { data: JsonLdNode | JsonLdNode[] }) { + const json = JSON.stringify(data).replace(/; +} diff --git a/apps/website/src/lib/structured-data.spec.ts b/apps/website/src/lib/structured-data.spec.ts new file mode 100644 index 000000000..3f6541734 --- /dev/null +++ b/apps/website/src/lib/structured-data.spec.ts @@ -0,0 +1,169 @@ +// SPDX-License-Identifier: MIT +import { describe, expect, it } from 'vitest'; +import { + breadcrumbJsonLd, + organizationJsonLd, + softwareSourceCodeJsonLd, + techArticleJsonLd, + websiteJsonLd, + blogPostingJsonLd, + type JsonLdNode, +} from './structured-data'; + +/** Every builder's output must survive a JSON round-trip unchanged. */ +function expectSerializable(data: JsonLdNode): Record { + const json = JSON.stringify(data); + expect(() => JSON.parse(json)).not.toThrow(); + return JSON.parse(json) as Record; +} + +describe('organizationJsonLd', () => { + it('describes Threadplane with an absolute url and logo', () => { + const data = organizationJsonLd(); + expect(data['@type']).toBe('Organization'); + expect(data['name']).toBe('Threadplane'); + expect(String(data['url'])).toBe('https://threadplane.ai/'); + expect(String(data['logo'])).toMatch(/^https:\/\/threadplane\.ai\//); + }); + + it('only links to profiles that exist', () => { + const sameAs = organizationJsonLd()['sameAs'] as string[]; + expect(sameAs).toContain('https://github.com/cacheplane/angular-agent-framework'); + for (const url of sameAs) expect(url).toMatch(/^https:\/\//); + }); + + it('serializes to JSON', () => { + expect(expectSerializable(organizationJsonLd())['@context']).toBe('https://schema.org'); + }); +}); + +describe('websiteJsonLd', () => { + it('is a WebSite node pointing at the origin', () => { + expect(websiteJsonLd()['@type']).toBe('WebSite'); + }); + + it('attributes the site to the Organization node', () => { + const publisher = websiteJsonLd()['publisher'] as Record; + expect(publisher['@id']).toBe(organizationJsonLd()['@id']); + }); + + it('serializes to JSON', () => { + expectSerializable(websiteJsonLd()); + }); +}); + +describe('blogPostingJsonLd', () => { + it('carries headline, dates, author, and absolute urls', () => { + const data = blogPostingJsonLd({ + title: 'A Post', + description: 'About things.', + slug: 'a-post', + datePublished: '2026-08-13', + authorName: 'Brian Love', + tags: ['angular'], + }); + expect(data['@type']).toBe('BlogPosting'); + expect(data['headline']).toBe('A Post'); + expect(data['datePublished']).toBe('2026-08-13'); + expect(data['dateModified']).toBe('2026-08-13'); + expect((data['author'] as Record)['name']).toBe('Brian Love'); + expect(String(data['url'])).toBe('https://threadplane.ai/blog/a-post'); + }); + + it('prefers an explicit dateModified when one is known', () => { + const data = blogPostingJsonLd({ + title: 'A Post', + description: 'About things.', + slug: 'a-post', + datePublished: '2026-08-13', + dateModified: '2026-08-19', + authorName: 'Brian Love', + }); + expect(data['dateModified']).toBe('2026-08-19'); + }); + + it('emits an absolute image url', () => { + const data = blogPostingJsonLd({ + title: 'A Post', + description: 'About things.', + slug: 'a-post', + datePublished: '2026-08-13', + authorName: 'Brian Love', + }); + expect(String(data['image'])).toMatch(/^https:\/\/threadplane\.ai\//); + }); + + it('serializes to JSON', () => { + expectSerializable( + blogPostingJsonLd({ + title: 'A Post', + description: 'About things.', + slug: 'a-post', + datePublished: '2026-08-13', + authorName: 'Brian Love', + }), + ); + }); +}); + +describe('techArticleJsonLd', () => { + it('describes a docs page', () => { + const data = techArticleJsonLd({ + title: 'Installation', + description: 'Install it.', + pathname: '/docs/chat/getting-started/installation', + }); + expect(data['@type']).toBe('TechArticle'); + expect(String(data['url'])).toBe('https://threadplane.ai/docs/chat/getting-started/installation'); + }); + + it('omits dateModified entirely when none is known', () => { + const data = techArticleJsonLd({ + title: 'Installation', + description: 'Install it.', + pathname: '/docs/chat/getting-started/installation', + }); + expect('dateModified' in data).toBe(false); + }); + + it('serializes to JSON', () => { + expectSerializable( + techArticleJsonLd({ title: 'Installation', description: 'Install it.', pathname: '/docs' }), + ); + }); +}); + +describe('breadcrumbJsonLd', () => { + it('numbers positions from 1 and resolves absolute urls', () => { + const data = breadcrumbJsonLd([ + { name: 'Docs', pathname: '/docs' }, + { name: 'Chat', pathname: '/docs/chat' }, + ]); + const items = data['itemListElement'] as Record[]; + expect(items).toHaveLength(2); + expect(items[0]['position']).toBe(1); + expect(String(items[1]['item'])).toBe('https://threadplane.ai/docs/chat'); + }); + + it('serializes to JSON', () => { + expectSerializable(breadcrumbJsonLd([{ name: 'Docs', pathname: '/docs' }])); + }); +}); + +describe('softwareSourceCodeJsonLd', () => { + it('marks Threadplane as an Angular TypeScript library', () => { + const data = softwareSourceCodeJsonLd(); + expect(data['@type']).toBe('SoftwareSourceCode'); + expect(data['programmingLanguage']).toBe('TypeScript'); + }); + + it('points at the real repository', () => { + expect(softwareSourceCodeJsonLd()['codeRepository']).toBe( + 'https://github.com/cacheplane/angular-agent-framework', + ); + }); + + it('serializes to JSON', () => { + expectSerializable(softwareSourceCodeJsonLd()); + }); +}); diff --git a/apps/website/src/lib/structured-data.ts b/apps/website/src/lib/structured-data.ts new file mode 100644 index 000000000..1ba001389 --- /dev/null +++ b/apps/website/src/lib/structured-data.ts @@ -0,0 +1,149 @@ +// SPDX-License-Identifier: MIT +import { DEFAULT_SOCIAL_IMAGE, getCanonicalUrl, SITE_NAME } from './site-metadata'; +import { SHORT_POSITIONING_DESCRIPTION } from './positioning'; + +/** A single schema.org node, ready to be serialized into a `ld+json` script. */ +export type JsonLdNode = Record; + +/** + * The canonical repository. Verified public; the npm *organization* page is + * member-gated, so `sameAs` links the public package page instead. + */ +const REPOSITORY_URL = 'https://github.com/cacheplane/angular-agent-framework'; + +/** + * Stable identity for the publisher node. Every other node refers to the + * Organization by `@id` rather than repeating it, which is the schema.org way + * to express "same entity" across nodes. + * + * COUPLING: those references only resolve if the Organization node itself is + * present in the page's structured data. It is mounted once in the root layout + * (task 8), so it is on every route; if that mount is ever removed, these + * references become dangling. + * + * Computed at module load, which is safe because {@link getCanonicalUrl} + * resolves against a hardcoded `SITE_ORIGIN` constant — no env lookup, no + * request context, identical on server and client. + */ +const ORGANIZATION_ID = `${getCanonicalUrl('/')}#organization`; + +export function organizationJsonLd(): JsonLdNode { + return { + '@context': 'https://schema.org', + '@type': 'Organization', + '@id': ORGANIZATION_ID, + name: SITE_NAME, + url: getCanonicalUrl('/'), + // No standalone brand mark exists yet (the in-app LogoMark is inline JSX), + // so this points at the generated site social card: a real, crawlable PNG + // carrying the wordmark. Swap in a dedicated mark when one ships. + logo: getCanonicalUrl(DEFAULT_SOCIAL_IMAGE), + description: + 'Threadplane builds the Angular UI layer for production agent applications on LangGraph and AG-UI-compatible runtimes.', + sameAs: [REPOSITORY_URL, 'https://www.npmjs.com/package/@threadplane/chat'], + }; +} + +export function websiteJsonLd(): JsonLdNode { + return { + '@context': 'https://schema.org', + '@type': 'WebSite', + '@id': `${getCanonicalUrl('/')}#website`, + name: SITE_NAME, + url: getCanonicalUrl('/'), + description: SHORT_POSITIONING_DESCRIPTION, + publisher: { '@id': ORGANIZATION_ID }, + }; +} + +export function softwareSourceCodeJsonLd(): JsonLdNode { + return { + '@context': 'https://schema.org', + '@type': 'SoftwareSourceCode', + name: '@threadplane/chat', + description: + 'Signal-native Angular chat UI primitives bound to a runtime-neutral Agent contract, with adapters for LangGraph and AG-UI.', + programmingLanguage: 'TypeScript', + runtimePlatform: 'Angular', + codeRepository: REPOSITORY_URL, + author: { '@id': ORGANIZATION_ID }, + license: getCanonicalUrl('/docs/licensing'), + }; +} + +/** The subset of a blog post that schema.org cares about. */ +export interface BlogPostingInput { + title: string; + description: string; + slug: string; + /** ISO 8601 publish date or timestamp. */ + datePublished: string; + /** ISO 8601 last-modified timestamp; falls back to `datePublished`. */ + dateModified?: string; + authorName: string; + tags?: string[]; +} + +export function blogPostingJsonLd(post: BlogPostingInput): JsonLdNode { + const url = getCanonicalUrl(`/blog/${post.slug}`); + return { + '@context': 'https://schema.org', + '@type': 'BlogPosting', + headline: post.title, + description: post.description, + url, + mainEntityOfPage: url, + datePublished: post.datePublished, + // Same "unmodified" rule the OpenGraph metadata uses: a post with no known + // modification advertises its publish date. + dateModified: post.dateModified ?? post.datePublished, + // TODO(task 9): switch to `/opengraph-image` once that route + // actually serves. It exists but currently 500s in production, and naming + // it here would advertise a broken image — same call task 6 made for + // og:image, kept consistent on purpose. + image: getCanonicalUrl(DEFAULT_SOCIAL_IMAGE), + keywords: post.tags, + // No `url` on the author until /about exists (task 11); a 404 author URL is + // worse than an unlinked name. + author: { '@type': 'Person', name: post.authorName }, + publisher: { '@id': ORGANIZATION_ID }, + }; +} + +/** The subset of a docs page that schema.org cares about. */ +export interface TechArticleInput { + title: string; + description: string; + pathname: string; + /** ISO 8601 last-modified timestamp; omitted from the node when unknown. */ + dateModified?: string; +} + +export function techArticleJsonLd(doc: TechArticleInput): JsonLdNode { + const url = getCanonicalUrl(doc.pathname); + return { + '@context': 'https://schema.org', + '@type': 'TechArticle', + headline: doc.title, + description: doc.description, + url, + mainEntityOfPage: url, + ...(doc.dateModified ? { dateModified: doc.dateModified } : {}), + author: { '@id': ORGANIZATION_ID }, + publisher: { '@id': ORGANIZATION_ID }, + proficiencyLevel: 'Expert', + }; +} + +export function breadcrumbJsonLd(crumbs: { name: string; pathname: string }[]): JsonLdNode { + return { + '@context': 'https://schema.org', + '@type': 'BreadcrumbList', + itemListElement: crumbs.map((crumb, index) => ({ + '@type': 'ListItem', + position: index + 1, + name: crumb.name, + item: getCanonicalUrl(crumb.pathname), + })), + }; +} From 4df555f0adaca8bc7b9352c294ee3a1631a28ee3 Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 20 Aug 2026 13:12:47 -0700 Subject: [PATCH 17/35] test(website): make json-ld round-trip assertions real expectSerializable claimed a round-trip but never compared the parsed result to the input, so JSON.parse(JSON.stringify(obj)) could not fail and the six "serializes to JSON" tests were vacuous. Adding toStrictEqual immediately caught blogPostingJsonLd leaving an undefined-valued `keywords` key when a post has no tags, where techArticleJsonLd already used the omit pattern for dateModified; both builders now agree. Also adds table-driven coverage of @context and the Organization @id reference across every builder that carries one, a rootJsonLd() @graph that makes the three root-layout nodes physically inseparable so an @id reference cannot be orphaned by mounting a subset, and an exported BreadcrumbCrumb type. Drops Organization.logo: a 1200x630 marketing social card is not a brand mark, and no square mark exists in the repo. Omitting is honest; the property should be restored once a real mark ships. Co-Authored-By: Claude Opus 5 --- apps/website/src/components/shared/JsonLd.tsx | 19 +- apps/website/src/lib/structured-data.spec.ts | 188 ++++++++++++------ apps/website/src/lib/structured-data.ts | 77 +++++-- 3 files changed, 198 insertions(+), 86 deletions(-) diff --git a/apps/website/src/components/shared/JsonLd.tsx b/apps/website/src/components/shared/JsonLd.tsx index 8b46e224a..6c4f0de7e 100644 --- a/apps/website/src/components/shared/JsonLd.tsx +++ b/apps/website/src/components/shared/JsonLd.tsx @@ -2,10 +2,21 @@ import type { JsonLdNode } from '../../lib/structured-data'; /** - * Renders schema.org JSON-LD. Content is generated from our own data, never - * from user input, so `dangerouslySetInnerHTML` is safe here; `<` is still - * escaped so a stray value cannot close the script tag. `<` is a valid - * JSON string escape, so the escaped payload still parses back identically. + * Renders schema.org JSON-LD. + * + * React does not escape the body of a `