Add Vary: Accept header and document nginx content negotiation config by mich-elle-luna · Pull Request #3548 · redis/docs

mich-elle-luna · 2026-06-24T21:43:31Z

Add Vary: Accept to all hugo serve responses so caches know responses vary by Accept header
Serve .html.md files with Content-Type: text/markdown during local dev
Document the production nginx config needed for HTTP-level content negotiation (Accept: text/markdown → serve index.html.md) in AI_AGENT_DEVELOPER_GUIDE.md

The element is already generated for every page by the AlternativeOutputFormats loop in baseof.html — no changes needed there.

Note

Low Risk
Build-time Hugo templates and local dev server headers only; no runtime app, auth, or data-path changes in this diff.

Overview
Extends AI/LLM alternate formats to the site home page and aligns local dev with Accept-based content negotiation.

config.toml now lists home in [outputs] with HTML, RSS, Markdown, and JSON (matching sections/pages). A trailing [server] block configures hugo serve only: Vary: Accept on all routes and Content-Type: text/markdown for **.html.md**, with a pointer to production nginx setup in the developer guide.

New layouts/home.md and layouts/home.json generate markdown (metadata block + processed body via process-markdown-content.html) and structured JSON (home metadata, processed content, and a children list of top-level pages with id/title/url/summary), following the same patterns as _default/section and single templates.

^{Reviewed by Cursor Bugbot for commit d92baf2. Bugbot is set up for automated code reviews on this repo. Configure here.}

- Add Vary: Accept to all hugo serve responses so caches know responses vary by Accept header - Serve .html.md files with Content-Type: text/markdown during local dev - Document the production nginx config needed for HTTP-level content negotiation (Accept: text/markdown → serve index.html.md) in AI_AGENT_DEVELOPER_GUIDE.md The <link rel="alternate" type="text/markdown"> element is already generated for every page by the AlternativeOutputFormats loop in baseof.html — no changes needed there. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

jit-ci · 2026-06-24T21:44:32Z

🛡️ Jit Security Scan Results

✅ No security findings were detected in this PR

^{Security scan by Jit}

andy-stark-redis · 2026-06-25T08:41:44Z

I don't know much about the server config but below is what the LLMs had to say about it. This might be a job for the new docs:assess-comments slash command, btw :-)

Content negotiation review — verdict

Two independent reviews (Claude + Codex) of the content-negotiation setup in config.toml and the nginx snippet in AI_AGENT_DEVELOPER_GUIDE.md. No files were changed. The Hugo output wiring and the dynamically generated <link rel="alternate" type="text/markdown"> (in layouts/_default/baseof.html) are sound; the issues are concentrated in output-kind coverage and the nginx example.

Agreed findings (both reviewers)

add_header Content-Type is the wrong directive (AI_AGENT_DEVELOPER_GUIDE.md ~L234). nginx sets MIME from types/default_type; .md isn't in the stock table, so the file is served as application/octet-stream and add_header may duplicate rather than replace it. Use default_type text/markdown; or a types { text/markdown md; } block.
Accept matching is a loose substring (~L227). It ignores q-values, so text/markdown;q=0 (explicit refusal) still gets served markdown, and it would match stray tokens. Not true negotiation.
Rewrite regex skips any path containing a dot (~L244). ^([^.]+)$ misses /…/index.html and version directories (e.g. 7.4), which fall through to HTML.

Caught by Codex (highest impact)

[outputs] omits the home kind (config.toml L125-127). Only section and page are listed, so the homepage (and any taxonomy/term pages) fall back to Hugo's default [HTML, RSS] and emit no .html.md or .json. This directly contradicts the guide's "every page generates a markdown version." Verified.
add_header … always flag missing (~L235, L239). Without always, nginx drops the Vary: Accept header on non-2xx/3xx responses (e.g. 404/5xx), weakening cache correctness.

Caught by Claude

Dev ≠ prod. The config.toml [server] block only sets headers — it does no Accept-based rewriting. hugo serve with Accept: text/markdown returns HTML, not markdown; the guide's wording invites the wrong assumption.
Verified the rel="alternate" claim genuinely holds (generated in baseof.html L7-9, not hardcoded).

Suggested priority

Add home (+ taxonomy/term if wanted) to [outputs] — otherwise the homepage has no markdown/JSON variant.
Fix the nginx Content-Type (use default_type/types, not add_header).
Add always to the Vary headers.
Harden the rewrite (dot-paths) and Accept match (q-values).
Note in the guide that the dev server does headers only, not negotiation.

Reviewed by Claude (Opus 4.8) and Codex; no code changed.

- Add home to [outputs] so the homepage and taxonomy pages emit .html.md and .json variants (previously fell back to [HTML, RSS]) - Fix nginx Content-Type: use types{} block instead of add_header so .md files get the correct MIME from the types table rather than a header that can't override nginx's own content-type - Add always flag to all Vary: Accept add_header directives so the header is sent on non-2xx responses (404/5xx) for cache correctness - Harden Accept map: exclude q=0 (explicit refusal) to avoid serving markdown to clients that don't want it - Fix rewrite regex: replace [^.]+ (skips dot-paths) with [^.]*[^/] so version directories (7.4/) and dotless paths are handled correctly - Clarify dev server limitation: hugo serve sets headers only and does not perform Accept-based rewriting; full negotiation requires nginx Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

paoloredis

We're going to need the following files for this change to work:

layouts/home.md
layouts/home.json

I'm not sure who the nginx instructions in the README are for. Is it for the agents?

mich-elle-luna · 2026-06-30T18:15:38Z

Hi thanks for reviewing this. It is for agents, but can't break anything for humans of course. This comes from the isitagentready.com reporting site by cloudflare, so if you run the checks on that site, this PR (and the other PR for the link header) come back as areas we can improve. So if we can do this and improve our score to show we can get the same score as these other hosted docs sites, that would be great.

- Add layouts/home.md and layouts/home.json so Hugo generates Markdown and JSON output formats for the homepage (config.toml lists both in home outputs but had no corresponding templates) - Remove nginx content negotiation section from AI_AGENT_DEVELOPER_GUIDE.md; production routing config belongs in ops/infra docs, not the AI agent guide Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

cursor

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit d92baf2. Configure here.}

cursor · 2026-06-30T18:23:26Z

+  {{- with .File -}}
+    {{- $baseName := .ContentBaseName -}}
+    {{- if or (not $baseName) (eq $baseName "_index") -}}
+      {{- $childId = $.Title | urlize -}}


Wrong child id in loop

Medium Severity

In the children loop, when a child’s ContentBaseName is missing or _index, id is set from $.Title (the home page) instead of the child page’s title. Top-level section entries can share the same wrong id in home index.json, which misleads consumers of the JSON/NDJSON feed.

^{Reviewed by Cursor Bugbot for commit d92baf2. Configure here.}

paoloredis · 2026-07-03T10:24:01Z

+page    = ["HTML", "Markdown", "JSON"]
+
+# Content negotiation headers (hugo serve only).
+# For production nginx config see AI_AGENT_DEVELOPER_GUIDE.md.


This comment is outdated now

paoloredis · 2026-07-03T11:41:02Z

I have now deployed a transform rule on Cloudflare so now the docs honor the Accept: text/markdown header:

$ curl -I https://redis.io/docs/latest/develop/get-started/ 
HTTP/2 200 
date: Fri, 03 Jul 2026 10:25:13 GMT
content-type: text/html
cf-ray: a1552f5bfd39b5d3-FCO
cf-cache-status: HIT
accept-ranges: bytes
age: 338
cache-control: public, max-age=14400
expires: Fri, 03 Jul 2026 14:25:13 GMT
last-modified: Thu, 02 Jul 2026 16:51:50 GMT
server: cloudflare
strict-transport-security: max-age=10368000; includeSubDomains; preload
x-goog-generation: 1783011110514563
x-goog-hash: crc32c=FelLtQ==
x-goog-hash: md5=jSixiSxxR7VAqBEclrkPPw==
x-goog-meta-goog-reserved-file-mtime: 1783011003
x-goog-metageneration: 1
x-goog-storage-class: STANDARD
x-goog-stored-content-encoding: identity
x-goog-stored-content-length: 81985
x-guploader-uploadid: AJ5rDhGVpc5SNvJXh04GDPeoMxz6DK9OQn6tDgL9OHaCYj_eZY_txgSSmINZqveCBuMcJyaf

$ curl -I https://redis.io/docs/latest/develop/get-started/ -H "Accept: text/markdown"
HTTP/2 200 
date: Fri, 03 Jul 2026 10:25:23 GMT
content-type: text/markdown
content-length: 912
cf-ray: a1552f97d978e1f2-FCO
cf-cache-status: HIT
accept-ranges: bytes
age: 356
cache-control: public, max-age=14400
etag: "54798ac25b774075200ecebc46555d4d"
expires: Fri, 03 Jul 2026 14:25:23 GMT
last-modified: Mon, 20 Apr 2026 13:52:45 GMT
server: cloudflare
strict-transport-security: max-age=10368000; includeSubDomains; preload
x-goog-generation: 1776693165498540
x-goog-hash: crc32c=fzJA1w==
x-goog-hash: md5=VHmKwlt3QHUgDs68RlVdTQ==
x-goog-meta-goog-reserved-file-mtime: 1776693004
x-goog-metageneration: 1
x-goog-storage-class: STANDARD
x-goog-stored-content-encoding: identity
x-goog-stored-content-length: 912
x-guploader-uploadid: AJ5rDhFg-X2BCzDaETenSFuDWqomspkkphkXyhAZYBQ7E1AlN0NVIalbCI4WDmDRmrFrjVXOea8h65M

paoloredis · 2026-07-03T13:16:06Z

isitagentready.com still reports Site does not support Markdown for Agents but that's because it looks at redis.io, as far as I can tell there's no way to have it run against a custom path, so I'd say we can ignore that suggestion with the changes I've made on Cloudflare

mich-elle-luna requested a review from a team June 24, 2026 21:47

mich-elle-luna requested a review from paoloredis June 25, 2026 22:42

cursor Bot reviewed Jun 25, 2026

View reviewed changes

Comment thread AI_AGENT_DEVELOPER_GUIDE.md Outdated

paoloredis requested changes Jun 30, 2026

View reviewed changes

cursor Bot reviewed Jun 30, 2026

View reviewed changes

paoloredis reviewed Jul 3, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add Vary: Accept header and document nginx content negotiation config#3548

Add Vary: Accept header and document nginx content negotiation config#3548
mich-elle-luna wants to merge 3 commits into
mainfrom
content-negotiation

mich-elle-luna commented Jun 24, 2026 •

edited by cursor Bot

Loading

Uh oh!

jit-ci Bot commented Jun 24, 2026

Uh oh!

andy-stark-redis commented Jun 25, 2026 •

edited

Loading

Uh oh!

Uh oh!

paoloredis left a comment

Uh oh!

mich-elle-luna commented Jun 30, 2026

Uh oh!

cursor Bot left a comment

Uh oh!

cursor Bot Jun 30, 2026

Uh oh!

paoloredis Jul 3, 2026

Uh oh!

paoloredis commented Jul 3, 2026

Uh oh!

paoloredis commented Jul 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

mich-elle-luna commented Jun 24, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jit-ci Bot commented Jun 24, 2026

🛡️ Jit Security Scan Results

Uh oh!

andy-stark-redis commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Content negotiation review — verdict

Agreed findings (both reviewers)

Caught by Codex (highest impact)

Caught by Claude

Suggested priority

Uh oh!

Uh oh!

paoloredis left a comment

Choose a reason for hiding this comment

Uh oh!

mich-elle-luna commented Jun 30, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot Jun 30, 2026

Choose a reason for hiding this comment

Wrong child id in loop

Uh oh!

paoloredis Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

paoloredis commented Jul 3, 2026

Uh oh!

paoloredis commented Jul 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mich-elle-luna commented Jun 24, 2026 •

edited by cursor Bot

Loading

andy-stark-redis commented Jun 25, 2026 •

edited

Loading