From e527a2b5ac247744524232e7f93bf2f5f8007439 Mon Sep 17 00:00:00 2001 From: Evan Hicks Date: Mon, 22 Jun 2026 12:07:19 -0400 Subject: [PATCH 1/9] feat: add create-kafka-topic skill Adds a skill that provisions a new Kafka topic across the four repos that must agree for a Sentry topic to exist: sentry-kafka-schemas (schema + CODEOWNERS), ops (partitions, per-region overrides, all_topics), sentry (Topic enum + cluster mapping), and getsentry (cellsilo cluster mapping). Opens one PR per repo and reports the four PR links with their merge-order dependencies. Reconciled against the taskworker-ingest-push topic PRs (sentry-kafka-schemas#484, ops#20917, sentry#117135, getsentry#20495). CI handles ops materialized config, so the skill only edits source-of-truth files. Co-Authored-By: Claude Opus 4.8 (1M context) --- .claude/settings.json | 1 + README.md | 1 + skills/claude-settings-audit/SKILL.md | 1 + skills/create-kafka-topic/SKILL.md | 214 ++++++++++++++++++++++++++ skills/create-kafka-topic/SPEC.md | 67 ++++++++ 5 files changed, 284 insertions(+) create mode 100644 skills/create-kafka-topic/SKILL.md create mode 100644 skills/create-kafka-topic/SPEC.md diff --git a/.claude/settings.json b/.claude/settings.json index 77faf37..1200ff9 100644 --- a/.claude/settings.json +++ b/.claude/settings.json @@ -46,6 +46,7 @@ "Skill(sentry-skills:code-simplifier)", "Skill(sentry-skills:commit)", "Skill(sentry-skills:create-branch)", + "Skill(sentry-skills:create-kafka-topic)", "Skill(sentry-skills:django-access-review)", "Skill(sentry-skills:django-perf-review)", "Skill(sentry-skills:doc-coauthoring)", diff --git a/README.md b/README.md index 3cfbf26..62aecb5 100644 --- a/README.md +++ b/README.md @@ -49,6 +49,7 @@ Works with Claude Code, Cursor, Cline, GitHub Copilot, and other compatible agen | [code-simplifier](skills/code-simplifier/SKILL.md) | Simplifies and refines code for clarity, consistency, and maintainability while preserving all functionality. | | [commit](skills/commit/SKILL.md) | ALWAYS use this skill when committing code changes — never commit directly without it. | | [create-branch](skills/create-branch/SKILL.md) | Create git branches following Sentry naming conventions. | +| [create-kafka-topic](skills/create-kafka-topic/SKILL.md) | Create a new Kafka topic across sentry-kafka-schemas, ops, sentry, and getsentry. | | [django-access-review](skills/django-access-review/SKILL.md) | Django access control and IDOR security review. | | [django-perf-review](skills/django-perf-review/SKILL.md) | Django performance code review. | | [doc-coauthoring](skills/doc-coauthoring/SKILL.md) | Guide users through a structured workflow for co-authoring documentation. | diff --git a/skills/claude-settings-audit/SKILL.md b/skills/claude-settings-audit/SKILL.md index 5da9bd2..cebfbd1 100644 --- a/skills/claude-settings-audit/SKILL.md +++ b/skills/claude-settings-audit/SKILL.md @@ -150,6 +150,7 @@ If this is a Sentry project (or sentry-skills plugin is installed), include: "Skill(sentry-skills:code-simplifier)", "Skill(sentry-skills:commit)", "Skill(sentry-skills:create-branch)", + "Skill(sentry-skills:create-kafka-topic)", "Skill(sentry-skills:django-access-review)", "Skill(sentry-skills:django-perf-review)", "Skill(sentry-skills:doc-coauthoring)", diff --git a/skills/create-kafka-topic/SKILL.md b/skills/create-kafka-topic/SKILL.md new file mode 100644 index 0000000..e51320d --- /dev/null +++ b/skills/create-kafka-topic/SKILL.md @@ -0,0 +1,214 @@ +--- +name: create-kafka-topic +description: Create a new Kafka topic across the Sentry stack — registers it in sentry-kafka-schemas, ops (shared_config/kafka), sentry, and getsentry, opening one PR per repo. Use when asked to "create a new Kafka topic", "add a Kafka topic", "register a Kafka topic", "set up a new Kafka topic", or provision a topic in sentry-kafka-schemas/ops/sentry/getsentry. +--- + +# Create a Kafka Topic + +Create and register a new Kafka topic across four repos, opening a PR in each: + +1. **sentry-kafka-schemas** — the topic schema definition + CODEOWNERS +2. **ops** — deployment config (default partitions + per-region overrides) +3. **sentry** — the `Topic` enum and cluster mapping +4. **getsentry** — the cell-silo topic→cluster mapping + +**Requires**: GitHub CLI (`gh`) authenticated, and local checkouts of all four repos. + +Run the steps in order. The end deliverable is **four PR links** returned to the user. + +## Step 0: Gather inputs and locate repos + +1. Prompt the user for: + - **Topic name** (kebab-case, e.g. `ingest-foo`) + - **Default number of partitions** + - **Owning team** (GitHub team handle, e.g. `@getsentry/taskbroker`) — used for CODEOWNERS +2. Locate each repo in the workspace (`sentry-kafka-schemas`, `ops`, `sentry`, `getsentry`). If any is not found near the working directory, ask the user for its path. Use `$SCHEMAS`, `$OPS`, `$SENTRY`, `$GETSENTRY` to refer to them below. +3. **Check for collision**: if `$SCHEMAS/topics/.yaml` already exists, tell the user the topic already exists and ask for a different name. Do not proceed until the name is free. + +Before each repo's work, ensure a clean tree on an updated default branch: + +```bash +cd "$REPO" +git stash list && git status --porcelain # must be clean; stop and ask if not +BASE=$(gh repo view --json defaultBranchRef --jq '.defaultBranchRef.name') +git checkout "$BASE" && git pull --ff-only +git checkout -b add-kafka-topic- +``` + +## Step 1: sentry-kafka-schemas PR + +Create `$SCHEMAS/topics/.yaml`. Read 2–3 existing files in `$SCHEMAS/topics/` that are closest to this topic's purpose to infer good values (pipeline, producer/consumer services, schema type, resource path). + +```yaml +pipeline: {pipeline_name} +description: | + {description} +services: + producers: + - {producer_service} + consumers: + - {consumer_service} +schemas: + - version: {version} + compatibility_mode: none + type: protobuf + resource: {resource_name} + examples: + - {example_path} +topic_creation_config: + compression.type: lz4 + message.timestamp.type: LogAppendTime + max.message.bytes: "10000000" + retention.ms: "86400000" +``` + +Field guidance: +- `services` use `getsentry/` form (e.g. `getsentry/sentry`, `getsentry/relay`). +- `type` is one of `protobuf`, `msgpack`, or `json`. `resource` and `examples` must point at a real schema (e.g. a `sentry_protos` class for protobuf, or a `*.schema.json` for json). If the schema does not exist yet, flag this to the user — the topic file references it. +- Keep the `topic_creation_config` block exactly as above unless the user requests different retention/size. + +Also add the topic to `$SCHEMAS/CODEOWNERS`, next to the sibling `topics/*` entries, using the owning team from Step 0: + +``` +/topics/.yaml +``` + +**Show the generated files to the user and ask if the values are correct.** If not, prompt for corrections and update the files. Then commit and open the PR: + +```bash +git add topics/.yaml CODEOWNERS +git commit -m "feat: add topic schema" +gh pr create --fill --title "feat: add topic schema" +``` + +Capture the PR URL. + +> **Release dependency**: the new topic only becomes usable downstream once a new `sentry-kafka-schemas` release is published (after this PR merges, per the repo's release process). The sentry and getsentry/ops dependency bumps in later steps reference that released version, so they may need to happen after this PR merges and releases. + +## Step 2: Choose regions, partitions, and clusters + +1. Determine the set of regions that get an override file. Do **not** assume every folder under `regional_overrides/`. Mirror a comparable existing topic — list which regions a sibling topic defines: + ```bash + for d in "$OPS"/shared_config/kafka/topics/regional_overrides/*/; do + [ -f "$d/.yaml" ] && basename "$d" + done + ``` + The `control` region typically does **not** carry these topics (it only has `taskworker-control*` topics) — exclude it unless the sibling topic includes it. +2. From that region set, prompt the user for **which regions the topic should be enabled in**. +3. For each **enabled** region, prompt for: + - **Number of partitions** in that region + - **Cluster** — present the valid choices for that region: + ```bash + grep -rh '^cluster:' "$OPS/shared_config/kafka/topics/regional_overrides//" | sort -u + ``` + +Regions not listed as enabled will be written as `disabled: true` in Step 3. + +## Step 3: ops PR + +In `$OPS`: + +1. **Default partitions** — create `shared_config/kafka/topics/.yaml`: + ```yaml + partitions: {default_partitions} + ``` +2. **Per-region overrides** — for **each region in the set from Step 2** (not every folder; `control` is normally excluded), create `regional_overrides//.yaml`: + - Region **not** enabled by the user → contents are exactly: + ```yaml + disabled: true + ``` + - Region enabled, partitions **differ** from the default: + ```yaml + cluster: {cluster_name} + partitions: {region_partitions} + ``` + - Region enabled, partitions **equal** the default → omit `partitions`, write only: + ```yaml + cluster: {cluster_name} + ``` +3. **Register for deployment** — add `` to the `all_deployed_topics:` list in `shared_config/kafka/topics/defaults/all_topics.yaml`. The list is grouped by topic family, not strictly alphabetical — insert it next to its sibling topics (e.g. right after ``). +4. **Bump the schemas dependency** — update the `sentry-kafka-schemas==` pin in `python/requirements.txt` to the version that includes the new topic (see the release dependency note in Step 1). + +> **Do not edit generated/materialized files.** CI regenerates `shared_config/_materialized_configs/`, `k8s/clusters/*/_topicctl_generated.yaml`, `k8s/materialized_manifests/`, and topicctl job manifests via `make materialize`. Only edit the source files above. + +Commit and open the PR: + +```bash +git add shared_config/kafka/topics/ python/requirements.txt +git commit -m "feat: deploy topic" +gh pr create --fill --title "feat: deploy topic" +``` + +Capture the PR URL. + +## Step 4: sentry PR + +In `$SENTRY`: + +1. **`src/sentry/conf/types/kafka_definition.py`** — add an entry to the `class Topic(Enum)`, matching the existing `SCREAMING_SNAKE_CASE = "kebab-name"` convention. Place it next to its sibling topics (the enum is grouped by family, not strictly alphabetical): + ```python + TOPIC_NAME = "" + ``` +2. **`src/sentry/conf/server.py`** — add an entry to the `KAFKA_TOPIC_TO_CLUSTER` dict with value `"default"`, next to the sibling entries: + ```python + "": "default", + ``` +3. **Bump the schemas dependency** — update `sentry-kafka-schemas>=` in `pyproject.toml` to the released version that includes the new topic, then regenerate the lock so `uv.lock` matches (`uv lock`). See the release dependency note in Step 1. + +Commit and open the PR: + +```bash +git add src/sentry/conf/types/kafka_definition.py src/sentry/conf/server.py pyproject.toml uv.lock +git commit -m "feat: register kafka topic" +gh pr create --fill --title "feat: register kafka topic" +``` + +Capture the PR URL. + +## Step 5: getsentry PR + +In `$GETSENTRY`, edit `getsentry/conf/settings/cellsilo.py`: + +Add an entry to the topic→cluster mapping that contains the other Kafka topics (find it by the sibling `taskworker-*`/topic entries). The value is a **real cluster name**, not `"default"` — use the same cluster the sibling topics in this dict use (e.g. `kafka-small`). Note the cluster name here can differ from the ops cluster name for the same region. + +```python +"": "", +``` + +Commit and open the PR: + +```bash +git add getsentry/conf/settings/cellsilo.py +git commit -m "feat: add topic" +gh pr create --fill --title "feat: add topic" +``` + +Capture the PR URL. + +## Step 6: Report + +Return all four PR links to the user **and the dependency ordering between them**, so the user knows what must merge first. The schemas PR must merge and publish a release before the dependency-bump PRs can reference the released version. + +Output in this shape (annotate each PR with its dependency): + +``` +Opened 4 PRs for the `` topic: + +1. sentry-kafka-schemas: + └─ Merge + release FIRST. Blocks #2 and #3 (they pin the new schemas version). +2. ops: + └─ Depends on #1's release (python/requirements.txt bump). +3. sentry: + └─ Depends on #1's release (pyproject.toml + uv.lock bump). +4. getsentry: + └─ Independent config; safe to merge anytime. + +Merge order: #1 (and its release) → then #2, #3, #4. +``` + +If a dependency bump was deferred because the schemas release was not yet published, say so explicitly and note which PR(s) still need the version updated. + +## Notes + +- For commit messages and PR descriptions, you may use the `commit` and `pr-writer` skills to follow Sentry conventions; the commands above are a self-contained fallback. +- The schemas PR (Step 1) gates the dependency bumps in ops (Step 3) and sentry (Step 4), which reference the released schemas version. The enum/config PRs and the getsentry PR are otherwise independent — open them even if one needs follow-up. diff --git a/skills/create-kafka-topic/SPEC.md b/skills/create-kafka-topic/SPEC.md new file mode 100644 index 0000000..94b419f --- /dev/null +++ b/skills/create-kafka-topic/SPEC.md @@ -0,0 +1,67 @@ +# Create Kafka Topic Specification + +## Intent + +The `create-kafka-topic` skill provisions a brand-new Kafka topic across the four repos that must all agree for a Sentry topic to exist and deploy: `sentry-kafka-schemas` (schema + ownership), `ops` (deployment/partition/region config), `sentry` (topic enum + cluster mapping), and `getsentry` (cell-silo cluster mapping). + +Its main job is to turn a topic name and a few deployment choices into one correct, reviewable PR per repo, matching the conventions of the topics that already exist, and to hand the user the resulting PR links. + +## Scope + +In scope: + +- Prompting for the topic name, default partition count, owning team, and per-region enablement/partitions/cluster. +- Creating the schema file and CODEOWNERS entry in `sentry-kafka-schemas`. +- Creating the default partition file, per-region override files, and `all_topics.yaml` entry in `ops`, plus the `sentry-kafka-schemas` requirement bump. +- Adding the `Topic` enum entry and `KAFKA_TOPIC_TO_CLUSTER` entry in `sentry`, plus the `sentry-kafka-schemas` dependency bump. +- Adding the topic→cluster entry in `getsentry`'s `cellsilo.py`. +- Opening one PR per repo and reporting all four URLs. + +Out of scope: + +- Editing generated/materialized config (CI regenerates ops `_materialized_configs/`, `k8s/`, and topicctl manifests via `make materialize`). +- Publishing the `sentry-kafka-schemas` release that the dependency bumps depend on. +- Writing producer/consumer application code or schema definitions themselves. +- Modifying or migrating existing topics. + +## Users And Trigger Context + +- Primary users: Sentry engineers (and coding agents) standing up a new Kafka topic. +- Common requests: "create a new Kafka topic", "add a Kafka topic", "register a Kafka topic", "set up a new Kafka topic". +- Should not trigger for: changing an existing topic's partitions/clusters, debugging consumers, or general Kafka questions. + +## Runtime Contract + +- Required first actions: prompt for topic name, default partitions, and owning team; locate all four repos; verify the topic name is not already taken in `sentry-kafka-schemas/topics/`. +- Per-repo precondition: a clean working tree on an updated default branch before creating the topic branch. +- Required outputs: four pull requests (one per repo) plus the four URLs returned to the user. +- Non-negotiable constraints: + - Do not edit generated/materialized files in `ops`. + - Per-region override contents follow the exact rules: `disabled: true` when not enabled; `cluster:` only when enabled and partitions equal the default; `cluster:` + `partitions:` when they differ. + - The `getsentry` cluster value is a real cluster name (e.g. `kafka-small`), never `"default"`, and may differ from the ops cluster name. + - Insert enum / `all_topics.yaml` entries next to sibling topics (grouped by family), not alphabetically. + - Confirm the generated schema file with the user before opening the first PR. +- Sequencing: the schemas PR gates the dependency bumps in `sentry`/`ops`, which reference the released schema version; these may need to follow the schemas release. +- Expected bundled files loaded at runtime: only `SKILL.md`. + +## Evaluation + +A run is correct when: + +- All four PRs are opened, each containing only the source-of-truth changes for its repo (no generated files). +- The schema YAML, region overrides, enum entry, and cluster mappings match the conventions of comparable existing topics. +- The user receives four working PR links. + +Reference example: the `taskworker-ingest-push` topic was created via getsentry/sentry-kafka-schemas#484, getsentry/ops#20917, getsentry/sentry#117135, and getsentry/getsentry#20495 — the skill should reproduce that same change set for an equivalent topic. + +## Limitations + +- Cannot determine the released `sentry-kafka-schemas` version until that release is published; the dependency bumps may be deferred. +- Relies on a sibling topic to infer the correct region set and good field values; an entirely novel topic shape may need extra user input. +- Does not run repo test suites or CI; reviewers/CI validate the generated config. + +## Maintenance + +- If the four-repo wiring changes (new repo, renamed config dict, new required file), update the corresponding step and this spec. +- If `ops` changes how regions or clusters are configured, update Steps 2–3. +- Keep the reference example current if the canonical topic-creation PR set changes. From 2739a04d55997be62674c36b15a5b6d0c234f109 Mon Sep 17 00:00:00 2001 From: Evan Hicks Date: Mon, 22 Jun 2026 13:03:45 -0400 Subject: [PATCH 2/9] fix: drop obsolete getsentry step from create-kafka-topic Testing the skill surfaced two issues: - getsentry no longer needs a per-topic change. It loads KAFKA_TOPIC_TO_CLUSTER at runtime from the topicctl-generated YAML mounted by ops (getsentry#20512, #20661), so the cellsilo.py edit is obsolete. The skill is now 3 repos, not 4. - Make the "which regions to enable" prompt an explicit blocking gate so it is never inferred from a reference topic. Updates SKILL.md, SPEC.md, and the README description accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 2 +- skills/create-kafka-topic/SKILL.md | 49 +++++++++--------------------- skills/create-kafka-topic/SPEC.md | 27 +++++++++------- 3 files changed, 31 insertions(+), 47 deletions(-) diff --git a/README.md b/README.md index 62aecb5..a594a32 100644 --- a/README.md +++ b/README.md @@ -49,7 +49,7 @@ Works with Claude Code, Cursor, Cline, GitHub Copilot, and other compatible agen | [code-simplifier](skills/code-simplifier/SKILL.md) | Simplifies and refines code for clarity, consistency, and maintainability while preserving all functionality. | | [commit](skills/commit/SKILL.md) | ALWAYS use this skill when committing code changes — never commit directly without it. | | [create-branch](skills/create-branch/SKILL.md) | Create git branches following Sentry naming conventions. | -| [create-kafka-topic](skills/create-kafka-topic/SKILL.md) | Create a new Kafka topic across sentry-kafka-schemas, ops, sentry, and getsentry. | +| [create-kafka-topic](skills/create-kafka-topic/SKILL.md) | Create a new Kafka topic across sentry-kafka-schemas, ops, and sentry. | | [django-access-review](skills/django-access-review/SKILL.md) | Django access control and IDOR security review. | | [django-perf-review](skills/django-perf-review/SKILL.md) | Django performance code review. | | [doc-coauthoring](skills/doc-coauthoring/SKILL.md) | Guide users through a structured workflow for co-authoring documentation. | diff --git a/skills/create-kafka-topic/SKILL.md b/skills/create-kafka-topic/SKILL.md index e51320d..b2710b4 100644 --- a/skills/create-kafka-topic/SKILL.md +++ b/skills/create-kafka-topic/SKILL.md @@ -1,20 +1,21 @@ --- name: create-kafka-topic -description: Create a new Kafka topic across the Sentry stack — registers it in sentry-kafka-schemas, ops (shared_config/kafka), sentry, and getsentry, opening one PR per repo. Use when asked to "create a new Kafka topic", "add a Kafka topic", "register a Kafka topic", "set up a new Kafka topic", or provision a topic in sentry-kafka-schemas/ops/sentry/getsentry. +description: Create a new Kafka topic across the Sentry stack — registers it in sentry-kafka-schemas, ops (shared_config/kafka), and sentry, opening one PR per repo. Use when asked to "create a new Kafka topic", "add a Kafka topic", "register a Kafka topic", "set up a new Kafka topic", or provision a topic in sentry-kafka-schemas/ops/sentry. --- # Create a Kafka Topic -Create and register a new Kafka topic across four repos, opening a PR in each: +Create and register a new Kafka topic across three repos, opening a PR in each: 1. **sentry-kafka-schemas** — the topic schema definition + CODEOWNERS 2. **ops** — deployment config (default partitions + per-region overrides) 3. **sentry** — the `Topic` enum and cluster mapping -4. **getsentry** — the cell-silo topic→cluster mapping -**Requires**: GitHub CLI (`gh`) authenticated, and local checkouts of all four repos. +**Requires**: GitHub CLI (`gh`) authenticated, and local checkouts of all three repos. -Run the steps in order. The end deliverable is **four PR links** returned to the user. +Run the steps in order. The end deliverable is **three PR links** returned to the user. + +> **Not getsentry.** getsentry no longer needs a per-topic change — it loads `KAFKA_TOPIC_TO_CLUSTER` at runtime from the topicctl-generated YAML that ops mounts (getsentry/getsentry#20512, #20661). Do not edit `cellsilo.py`. ## Step 0: Gather inputs and locate repos @@ -22,7 +23,7 @@ Run the steps in order. The end deliverable is **four PR links** returned to the - **Topic name** (kebab-case, e.g. `ingest-foo`) - **Default number of partitions** - **Owning team** (GitHub team handle, e.g. `@getsentry/taskbroker`) — used for CODEOWNERS -2. Locate each repo in the workspace (`sentry-kafka-schemas`, `ops`, `sentry`, `getsentry`). If any is not found near the working directory, ask the user for its path. Use `$SCHEMAS`, `$OPS`, `$SENTRY`, `$GETSENTRY` to refer to them below. +2. Locate each repo in the workspace (`sentry-kafka-schemas`, `ops`, `sentry`). If any is not found near the working directory, ask the user for its path. Use `$SCHEMAS`, `$OPS`, `$SENTRY` to refer to them below. 3. **Check for collision**: if `$SCHEMAS/topics/.yaml` already exists, tell the user the topic already exists and ask for a different name. Do not proceed until the name is free. Before each repo's work, ensure a clean tree on an updated default branch: @@ -83,7 +84,7 @@ gh pr create --fill --title "feat: add topic schema" Capture the PR URL. -> **Release dependency**: the new topic only becomes usable downstream once a new `sentry-kafka-schemas` release is published (after this PR merges, per the repo's release process). The sentry and getsentry/ops dependency bumps in later steps reference that released version, so they may need to happen after this PR merges and releases. +> **Release dependency**: the new topic only becomes usable downstream once a new `sentry-kafka-schemas` release is published (after this PR merges, per the repo's release process). The sentry and ops dependency bumps in later steps reference that released version, so they may need to happen after this PR merges and releases. ## Step 2: Choose regions, partitions, and clusters @@ -94,7 +95,7 @@ Capture the PR URL. done ``` The `control` region typically does **not** carry these topics (it only has `taskworker-control*` topics) — exclude it unless the sibling topic includes it. -2. From that region set, prompt the user for **which regions the topic should be enabled in**. +2. **STOP and ask the user which regions the topic should be enabled in.** This is a required gate — never infer the enabled set from a reference topic, even when the user said "use the same values as ``" (that covers partitions/cluster/schema defaults, not enablement). Wait for an explicit answer before continuing. 3. For each **enabled** region, prompt for: - **Number of partitions** in that region - **Cluster** — present the valid choices for that region: @@ -165,34 +166,14 @@ gh pr create --fill --title "feat: register kafka topic" Capture the PR URL. -## Step 5: getsentry PR - -In `$GETSENTRY`, edit `getsentry/conf/settings/cellsilo.py`: - -Add an entry to the topic→cluster mapping that contains the other Kafka topics (find it by the sibling `taskworker-*`/topic entries). The value is a **real cluster name**, not `"default"` — use the same cluster the sibling topics in this dict use (e.g. `kafka-small`). Note the cluster name here can differ from the ops cluster name for the same region. - -```python -"": "", -``` - -Commit and open the PR: - -```bash -git add getsentry/conf/settings/cellsilo.py -git commit -m "feat: add topic" -gh pr create --fill --title "feat: add topic" -``` - -Capture the PR URL. - -## Step 6: Report +## Step 5: Report -Return all four PR links to the user **and the dependency ordering between them**, so the user knows what must merge first. The schemas PR must merge and publish a release before the dependency-bump PRs can reference the released version. +Return all three PR links to the user **and the dependency ordering between them**, so the user knows what must merge first. The schemas PR must merge and publish a release before the dependency-bump PRs can reference the released version. Output in this shape (annotate each PR with its dependency): ``` -Opened 4 PRs for the `` topic: +Opened 3 PRs for the `` topic: 1. sentry-kafka-schemas: └─ Merge + release FIRST. Blocks #2 and #3 (they pin the new schemas version). @@ -200,10 +181,8 @@ Opened 4 PRs for the `` topic: └─ Depends on #1's release (python/requirements.txt bump). 3. sentry: └─ Depends on #1's release (pyproject.toml + uv.lock bump). -4. getsentry: - └─ Independent config; safe to merge anytime. -Merge order: #1 (and its release) → then #2, #3, #4. +Merge order: #1 (and its release) → then #2 and #3. ``` If a dependency bump was deferred because the schemas release was not yet published, say so explicitly and note which PR(s) still need the version updated. @@ -211,4 +190,4 @@ If a dependency bump was deferred because the schemas release was not yet publis ## Notes - For commit messages and PR descriptions, you may use the `commit` and `pr-writer` skills to follow Sentry conventions; the commands above are a self-contained fallback. -- The schemas PR (Step 1) gates the dependency bumps in ops (Step 3) and sentry (Step 4), which reference the released schemas version. The enum/config PRs and the getsentry PR are otherwise independent — open them even if one needs follow-up. +- The schemas PR (Step 1) gates the dependency bumps in ops (Step 3) and sentry (Step 4), which reference the released schemas version. If the release is not yet published, open the ops/sentry PRs without the bump and flag it as a follow-up. diff --git a/skills/create-kafka-topic/SPEC.md b/skills/create-kafka-topic/SPEC.md index 94b419f..3389b05 100644 --- a/skills/create-kafka-topic/SPEC.md +++ b/skills/create-kafka-topic/SPEC.md @@ -2,7 +2,9 @@ ## Intent -The `create-kafka-topic` skill provisions a brand-new Kafka topic across the four repos that must all agree for a Sentry topic to exist and deploy: `sentry-kafka-schemas` (schema + ownership), `ops` (deployment/partition/region config), `sentry` (topic enum + cluster mapping), and `getsentry` (cell-silo cluster mapping). +The `create-kafka-topic` skill provisions a brand-new Kafka topic across the three repos that must all agree for a Sentry topic to exist and deploy: `sentry-kafka-schemas` (schema + ownership), `ops` (deployment/partition/region config), and `sentry` (topic enum + cluster mapping). + +`getsentry` is intentionally **not** in scope: it no longer maintains a per-topic cluster map by hand — it loads `KAFKA_TOPIC_TO_CLUSTER` at runtime from the topicctl-generated YAML mounted by ops (getsentry/getsentry#20512, #20661). Its main job is to turn a topic name and a few deployment choices into one correct, reviewable PR per repo, matching the conventions of the topics that already exist, and to hand the user the resulting PR links. @@ -14,11 +16,11 @@ In scope: - Creating the schema file and CODEOWNERS entry in `sentry-kafka-schemas`. - Creating the default partition file, per-region override files, and `all_topics.yaml` entry in `ops`, plus the `sentry-kafka-schemas` requirement bump. - Adding the `Topic` enum entry and `KAFKA_TOPIC_TO_CLUSTER` entry in `sentry`, plus the `sentry-kafka-schemas` dependency bump. -- Adding the topic→cluster entry in `getsentry`'s `cellsilo.py`. -- Opening one PR per repo and reporting all four URLs. +- Opening one PR per repo and reporting all three URLs. Out of scope: +- Editing `getsentry` config (it derives the cluster map from the ops-generated YAML at runtime; no per-topic change is needed). - Editing generated/materialized config (CI regenerates ops `_materialized_configs/`, `k8s/`, and topicctl manifests via `make materialize`). - Publishing the `sentry-kafka-schemas` release that the dependency bumps depend on. - Writing producer/consumer application code or schema definitions themselves. @@ -32,13 +34,13 @@ Out of scope: ## Runtime Contract -- Required first actions: prompt for topic name, default partitions, and owning team; locate all four repos; verify the topic name is not already taken in `sentry-kafka-schemas/topics/`. +- Required first actions: prompt for topic name, default partitions, and owning team; locate all three repos; verify the topic name is not already taken in `sentry-kafka-schemas/topics/`. - Per-repo precondition: a clean working tree on an updated default branch before creating the topic branch. -- Required outputs: four pull requests (one per repo) plus the four URLs returned to the user. +- Required outputs: three pull requests (one per repo) plus the three URLs returned to the user. - Non-negotiable constraints: - - Do not edit generated/materialized files in `ops`. + - Always ask the user which regions to enable; never infer the enabled set from a reference topic, even when told to "use the same values as ``". + - Do not edit generated/materialized files in `ops`, and do not edit `getsentry`. - Per-region override contents follow the exact rules: `disabled: true` when not enabled; `cluster:` only when enabled and partitions equal the default; `cluster:` + `partitions:` when they differ. - - The `getsentry` cluster value is a real cluster name (e.g. `kafka-small`), never `"default"`, and may differ from the ops cluster name. - Insert enum / `all_topics.yaml` entries next to sibling topics (grouped by family), not alphabetically. - Confirm the generated schema file with the user before opening the first PR. - Sequencing: the schemas PR gates the dependency bumps in `sentry`/`ops`, which reference the released schema version; these may need to follow the schemas release. @@ -48,11 +50,14 @@ Out of scope: A run is correct when: -- All four PRs are opened, each containing only the source-of-truth changes for its repo (no generated files). +- All three PRs are opened, each containing only the source-of-truth changes for its repo (no generated files). - The schema YAML, region overrides, enum entry, and cluster mappings match the conventions of comparable existing topics. -- The user receives four working PR links. +- The enabled-region set matches what the user explicitly chose (not a reference topic's set). +- The user receives three working PR links. -Reference example: the `taskworker-ingest-push` topic was created via getsentry/sentry-kafka-schemas#484, getsentry/ops#20917, getsentry/sentry#117135, and getsentry/getsentry#20495 — the skill should reproduce that same change set for an equivalent topic. +Reference examples: +- `taskworker-ingest-push` — getsentry/sentry-kafka-schemas#484, getsentry/ops#20917, getsentry/sentry#117135 (its getsentry/getsentry#20495 predates the loader change and is no longer needed). +- `taskworker-launchpad-push` — getsentry/sentry-kafka-schemas#490, getsentry/ops#21361, getsentry/sentry#118161 (s4s2-only enablement, schemas-release-deferred version bumps). ## Limitations @@ -62,6 +67,6 @@ Reference example: the `taskworker-ingest-push` topic was created via getsentry/ ## Maintenance -- If the four-repo wiring changes (new repo, renamed config dict, new required file), update the corresponding step and this spec. +- If the three-repo wiring changes (new repo, renamed config dict, new required file, or getsentry again needing a per-topic edit), update the corresponding step and this spec. - If `ops` changes how regions or clusters are configured, update Steps 2–3. - Keep the reference example current if the canonical topic-creation PR set changes. From f8a349ddc107795a12e962424805a56e61d6dea1 Mon Sep 17 00:00:00 2001 From: Evan Hicks Date: Mon, 22 Jun 2026 13:08:38 -0400 Subject: [PATCH 3/9] feat: request owning team as reviewer on all create-kafka-topic PRs The team used for the sentry-kafka-schemas CODEOWNERS entry is now also added as a reviewer (--reviewer) on all three PRs. Co-Authored-By: Claude Opus 4.8 (1M context) --- skills/create-kafka-topic/SKILL.md | 8 ++++---- skills/create-kafka-topic/SPEC.md | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/skills/create-kafka-topic/SKILL.md b/skills/create-kafka-topic/SKILL.md index b2710b4..521c063 100644 --- a/skills/create-kafka-topic/SKILL.md +++ b/skills/create-kafka-topic/SKILL.md @@ -22,7 +22,7 @@ Run the steps in order. The end deliverable is **three PR links** returned to th 1. Prompt the user for: - **Topic name** (kebab-case, e.g. `ingest-foo`) - **Default number of partitions** - - **Owning team** (GitHub team handle, e.g. `@getsentry/taskbroker`) — used for CODEOWNERS + - **Owning team** (GitHub team handle, e.g. `@getsentry/taskbroker`) — used for the `sentry-kafka-schemas` CODEOWNERS entry and requested as a **reviewer on all three PRs**. For `gh pr create --reviewer`, pass it as the `org/team` slug **without** the leading `@` (e.g. `getsentry/taskbroker`) — referred to as `` below. 2. Locate each repo in the workspace (`sentry-kafka-schemas`, `ops`, `sentry`). If any is not found near the working directory, ask the user for its path. Use `$SCHEMAS`, `$OPS`, `$SENTRY` to refer to them below. 3. **Check for collision**: if `$SCHEMAS/topics/.yaml` already exists, tell the user the topic already exists and ask for a different name. Do not proceed until the name is free. @@ -79,7 +79,7 @@ Also add the topic to `$SCHEMAS/CODEOWNERS`, next to the sibling `topics/*` entr ```bash git add topics/.yaml CODEOWNERS git commit -m "feat: add topic schema" -gh pr create --fill --title "feat: add topic schema" +gh pr create --fill --reviewer --title "feat: add topic schema" ``` Capture the PR URL. @@ -137,7 +137,7 @@ Commit and open the PR: ```bash git add shared_config/kafka/topics/ python/requirements.txt git commit -m "feat: deploy topic" -gh pr create --fill --title "feat: deploy topic" +gh pr create --fill --reviewer --title "feat: deploy topic" ``` Capture the PR URL. @@ -161,7 +161,7 @@ Commit and open the PR: ```bash git add src/sentry/conf/types/kafka_definition.py src/sentry/conf/server.py pyproject.toml uv.lock git commit -m "feat: register kafka topic" -gh pr create --fill --title "feat: register kafka topic" +gh pr create --fill --reviewer --title "feat: register kafka topic" ``` Capture the PR URL. diff --git a/skills/create-kafka-topic/SPEC.md b/skills/create-kafka-topic/SPEC.md index 3389b05..303c065 100644 --- a/skills/create-kafka-topic/SPEC.md +++ b/skills/create-kafka-topic/SPEC.md @@ -36,7 +36,7 @@ Out of scope: - Required first actions: prompt for topic name, default partitions, and owning team; locate all three repos; verify the topic name is not already taken in `sentry-kafka-schemas/topics/`. - Per-repo precondition: a clean working tree on an updated default branch before creating the topic branch. -- Required outputs: three pull requests (one per repo) plus the three URLs returned to the user. +- Required outputs: three pull requests (one per repo), each with the owning team requested as a reviewer, plus the three URLs returned to the user. - Non-negotiable constraints: - Always ask the user which regions to enable; never infer the enabled set from a reference topic, even when told to "use the same values as ``". - Do not edit generated/materialized files in `ops`, and do not edit `getsentry`. From 8a5b3a4ed650fd4694237cc52a47ca59a34a87a7 Mon Sep 17 00:00:00 2001 From: Evan Hicks Date: Mon, 22 Jun 2026 15:40:24 -0400 Subject: [PATCH 4/9] feat: add cookiecutter-region template step to create-kafka-topic New ops regional override files must have a counterpart in the cookiecutter-region template (disabled: true) so newly-created regions get the topic too; Warden's cookiecutter-region-backport check flags the omission. Co-Authored-By: Claude Opus 4.8 (1M context) --- skills/create-kafka-topic/SKILL.md | 11 ++++++++--- skills/create-kafka-topic/SPEC.md | 2 +- 2 files changed, 9 insertions(+), 4 deletions(-) diff --git a/skills/create-kafka-topic/SKILL.md b/skills/create-kafka-topic/SKILL.md index 521c063..d6727e4 100644 --- a/skills/create-kafka-topic/SKILL.md +++ b/skills/create-kafka-topic/SKILL.md @@ -127,15 +127,20 @@ In `$OPS`: ```yaml cluster: {cluster_name} ``` -3. **Register for deployment** — add `` to the `all_deployed_topics:` list in `shared_config/kafka/topics/defaults/all_topics.yaml`. The list is grouped by topic family, not strictly alphabetical — insert it next to its sibling topics (e.g. right after ``). -4. **Bump the schemas dependency** — update the `sentry-kafka-schemas==` pin in `python/requirements.txt` to the version that includes the new topic (see the release dependency note in Step 1). +3. **Cookiecutter region template** — also create the regional override in the new-region template so future regions get the topic: `cookiecutters/cookiecutter-region/shared_config/kafka/topics/regional_overrides/{{region}}/.yaml`. New regions start disabled, so its contents are exactly: + ```yaml + disabled: true + ``` + (Skipping this is flagged by Warden's `cookiecutter-region-backport` check — see the `taskworker-seer-push.yaml` precedent.) +4. **Register for deployment** — add `` to the `all_deployed_topics:` list in `shared_config/kafka/topics/defaults/all_topics.yaml`. The list is grouped by topic family, not strictly alphabetical — insert it next to its sibling topics (e.g. right after ``). +5. **Bump the schemas dependency** — update the `sentry-kafka-schemas==` pin in `python/requirements.txt` to the version that includes the new topic (see the release dependency note in Step 1). > **Do not edit generated/materialized files.** CI regenerates `shared_config/_materialized_configs/`, `k8s/clusters/*/_topicctl_generated.yaml`, `k8s/materialized_manifests/`, and topicctl job manifests via `make materialize`. Only edit the source files above. Commit and open the PR: ```bash -git add shared_config/kafka/topics/ python/requirements.txt +git add shared_config/kafka/topics/ cookiecutters/cookiecutter-region/ python/requirements.txt git commit -m "feat: deploy topic" gh pr create --fill --reviewer --title "feat: deploy topic" ``` diff --git a/skills/create-kafka-topic/SPEC.md b/skills/create-kafka-topic/SPEC.md index 303c065..1ea760d 100644 --- a/skills/create-kafka-topic/SPEC.md +++ b/skills/create-kafka-topic/SPEC.md @@ -14,7 +14,7 @@ In scope: - Prompting for the topic name, default partition count, owning team, and per-region enablement/partitions/cluster. - Creating the schema file and CODEOWNERS entry in `sentry-kafka-schemas`. -- Creating the default partition file, per-region override files, and `all_topics.yaml` entry in `ops`, plus the `sentry-kafka-schemas` requirement bump. +- Creating the default partition file, per-region override files, the cookiecutter-region template override (`disabled: true`), and `all_topics.yaml` entry in `ops`, plus the `sentry-kafka-schemas` requirement bump. - Adding the `Topic` enum entry and `KAFKA_TOPIC_TO_CLUSTER` entry in `sentry`, plus the `sentry-kafka-schemas` dependency bump. - Opening one PR per repo and reporting all three URLs. From eafad8e25ad688ae9cf2904b9f5342b6d19162ab Mon Sep 17 00:00:00 2001 From: Evan Hicks Date: Tue, 23 Jun 2026 14:33:15 -0400 Subject: [PATCH 5/9] feat: bump schemas pin to anticipated version in create-kafka-topic Instead of deferring the sentry-kafka-schemas pin bump, the ops and sentry PRs now bump to the anticipated next version (latest released patch + 1) so the required change is visible in the diff. The PR body flags the version as anticipated, and for sentry notes that uv.lock's resolved entry must be regenerated with `uv lock` once the release publishes. Co-Authored-By: Claude Opus 4.8 (1M context) --- skills/create-kafka-topic/SKILL.md | 10 ++++++---- skills/create-kafka-topic/SPEC.md | 4 ++-- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/skills/create-kafka-topic/SKILL.md b/skills/create-kafka-topic/SKILL.md index d6727e4..46faaf7 100644 --- a/skills/create-kafka-topic/SKILL.md +++ b/skills/create-kafka-topic/SKILL.md @@ -84,7 +84,9 @@ gh pr create --fill --reviewer --title "feat: add **Release dependency**: the new topic only becomes usable downstream once a new `sentry-kafka-schemas` release is published (after this PR merges, per the repo's release process). The sentry and ops dependency bumps in later steps reference that released version, so they may need to happen after this PR merges and releases. +> **Release dependency**: the new topic only becomes usable downstream once a new `sentry-kafka-schemas` release is published (after this PR merges, per the repo's release process). The ops and sentry PRs still **bump the pin now** to the anticipated next version so the required change is visible in the diff — see Steps 3 and 4. +> +> Compute the anticipated version: take the latest released version and increment the patch (`git -C $SCHEMAS tag --sort=-v:refname | head -1` → e.g. `2.1.35` → `2.1.36`). Releases are manually versioned, so always note in the PR that this is the *anticipated* version and must be confirmed/adjusted to match the actual published release before merge. Call this `` below. ## Step 2: Choose regions, partitions, and clusters @@ -133,7 +135,7 @@ In `$OPS`: ``` (Skipping this is flagged by Warden's `cookiecutter-region-backport` check — see the `taskworker-seer-push.yaml` precedent.) 4. **Register for deployment** — add `` to the `all_deployed_topics:` list in `shared_config/kafka/topics/defaults/all_topics.yaml`. The list is grouped by topic family, not strictly alphabetical — insert it next to its sibling topics (e.g. right after ``). -5. **Bump the schemas dependency** — update the `sentry-kafka-schemas==` pin in `python/requirements.txt` to the version that includes the new topic (see the release dependency note in Step 1). +5. **Bump the schemas dependency** — update the `sentry-kafka-schemas==` pin in `python/requirements.txt` to `` (see the release-dependency note in Step 1). The pin lives only in the compiled `requirements.txt`, not `requirements.in`. This is a clean, complete change (no hashes in this file). > **Do not edit generated/materialized files.** CI regenerates `shared_config/_materialized_configs/`, `k8s/clusters/*/_topicctl_generated.yaml`, `k8s/materialized_manifests/`, and topicctl job manifests via `make materialize`. Only edit the source files above. @@ -159,7 +161,7 @@ In `$SENTRY`: ```python "": "default", ``` -3. **Bump the schemas dependency** — update `sentry-kafka-schemas>=` in `pyproject.toml` to the released version that includes the new topic, then regenerate the lock so `uv.lock` matches (`uv lock`). See the release dependency note in Step 1. +3. **Bump the schemas dependency** — update `sentry-kafka-schemas>=` in `pyproject.toml` to ``, and bump the matching `specifier = ">=..."` line for `sentry-kafka-schemas` in `uv.lock` so the two agree. If `` is already published, run `uv lock` to fully regenerate. If it is **not** published yet (the usual case when this PR precedes the schemas release), you cannot regenerate `uv.lock`'s resolved version + wheel hash — leave the `[[package]]` `version`/`wheels` block as-is and note in the PR that `uv lock` must be re-run once the release publishes. See the release-dependency note in Step 1. Commit and open the PR: @@ -195,4 +197,4 @@ If a dependency bump was deferred because the schemas release was not yet publis ## Notes - For commit messages and PR descriptions, you may use the `commit` and `pr-writer` skills to follow Sentry conventions; the commands above are a self-contained fallback. -- The schemas PR (Step 1) gates the dependency bumps in ops (Step 3) and sentry (Step 4), which reference the released schemas version. If the release is not yet published, open the ops/sentry PRs without the bump and flag it as a follow-up. +- The schemas PR (Step 1) gates the dependency bumps in ops (Step 3) and sentry (Step 4). Bump the pins to the anticipated `` anyway so reviewers see the required change; flag in each PR body that the version is anticipated (confirm against the actual release) and, for sentry, that `uv.lock` needs `uv lock` once the release publishes. CI on those PRs may stay red until then. diff --git a/skills/create-kafka-topic/SPEC.md b/skills/create-kafka-topic/SPEC.md index 1ea760d..f5809fd 100644 --- a/skills/create-kafka-topic/SPEC.md +++ b/skills/create-kafka-topic/SPEC.md @@ -43,7 +43,7 @@ Out of scope: - Per-region override contents follow the exact rules: `disabled: true` when not enabled; `cluster:` only when enabled and partitions equal the default; `cluster:` + `partitions:` when they differ. - Insert enum / `all_topics.yaml` entries next to sibling topics (grouped by family), not alphabetically. - Confirm the generated schema file with the user before opening the first PR. -- Sequencing: the schemas PR gates the dependency bumps in `sentry`/`ops`, which reference the released schema version; these may need to follow the schemas release. +- Sequencing: the schemas PR gates the dependency bumps in `sentry`/`ops`. The ops/sentry PRs still bump the pin to the anticipated next version (latest released patch + 1) so the required change is visible; the PR body must flag it as anticipated and, for sentry, note that `uv.lock` must be regenerated with `uv lock` once the release publishes. - Expected bundled files loaded at runtime: only `SKILL.md`. ## Evaluation @@ -61,7 +61,7 @@ Reference examples: ## Limitations -- Cannot determine the released `sentry-kafka-schemas` version until that release is published; the dependency bumps may be deferred. +- The exact published `sentry-kafka-schemas` version is not known until release (versioning is manual); the skill bumps to the anticipated patch increment, which must be confirmed against the actual release. Sentry's `uv.lock` resolved entry + wheel hash cannot be regenerated until the release publishes. - Relies on a sibling topic to infer the correct region set and good field values; an entirely novel topic shape may need extra user input. - Does not run repo test suites or CI; reviewers/CI validate the generated config. From 6daf9019f4fdfed0d45cffe561940dcc9e96d769 Mon Sep 17 00:00:00 2001 From: Evan Hicks Date: Tue, 23 Jun 2026 14:37:51 -0400 Subject: [PATCH 6/9] fix: address review feedback on create-kafka-topic MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Enabled regions outside the sibling-derived candidate set are now always given an override file (file set = candidate ∪ user-enabled regions). - Use the full shared_config/kafka/topics/regional_overrides/... path for per-region override files so they're staged by git add. - Clarify the Topic enum member is SCREAMING_SNAKE_CASE while the value is the kebab-case topic name, with a concrete example. - Note the KAFKA_TOPIC_TO_CLUSTER value is "default" today, but to match a sibling family if that ever differs. Co-Authored-By: Claude Opus 4.8 (1M context) --- skills/create-kafka-topic/SKILL.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/skills/create-kafka-topic/SKILL.md b/skills/create-kafka-topic/SKILL.md index 46faaf7..5f15749 100644 --- a/skills/create-kafka-topic/SKILL.md +++ b/skills/create-kafka-topic/SKILL.md @@ -90,7 +90,7 @@ Capture the PR URL. ## Step 2: Choose regions, partitions, and clusters -1. Determine the set of regions that get an override file. Do **not** assume every folder under `regional_overrides/`. Mirror a comparable existing topic — list which regions a sibling topic defines: +1. Build the **candidate region set**. Do **not** assume every folder under `regional_overrides/`. Start from a comparable existing topic — list which regions a sibling topic defines: ```bash for d in "$OPS"/shared_config/kafka/topics/regional_overrides/*/; do [ -f "$d/.yaml" ] && basename "$d" @@ -98,14 +98,14 @@ Capture the PR URL. ``` The `control` region typically does **not** carry these topics (it only has `taskworker-control*` topics) — exclude it unless the sibling topic includes it. 2. **STOP and ask the user which regions the topic should be enabled in.** This is a required gate — never infer the enabled set from a reference topic, even when the user said "use the same values as ``" (that covers partitions/cluster/schema defaults, not enablement). Wait for an explicit answer before continuing. -3. For each **enabled** region, prompt for: +3. The **file set** — regions that get an override file — is the candidate set **plus every region the user enabled** (an enabled region must always get a file, even if the sibling topic did not define it). For each **enabled** region, prompt for: - **Number of partitions** in that region - **Cluster** — present the valid choices for that region: ```bash grep -rh '^cluster:' "$OPS/shared_config/kafka/topics/regional_overrides//" | sort -u ``` -Regions not listed as enabled will be written as `disabled: true` in Step 3. +Every region in the file set that is **not** enabled gets `disabled: true` (Step 3). ## Step 3: ops PR @@ -115,7 +115,7 @@ In `$OPS`: ```yaml partitions: {default_partitions} ``` -2. **Per-region overrides** — for **each region in the set from Step 2** (not every folder; `control` is normally excluded), create `regional_overrides//.yaml`: +2. **Per-region overrides** — for **each region in the file set from Step 2** (not every folder; `control` is normally excluded), create `shared_config/kafka/topics/regional_overrides//.yaml`: - Region **not** enabled by the user → contents are exactly: ```yaml disabled: true @@ -153,11 +153,12 @@ Capture the PR URL. In `$SENTRY`: -1. **`src/sentry/conf/types/kafka_definition.py`** — add an entry to the `class Topic(Enum)`, matching the existing `SCREAMING_SNAKE_CASE = "kebab-name"` convention. Place it next to its sibling topics (the enum is grouped by family, not strictly alphabetical): +1. **`src/sentry/conf/types/kafka_definition.py`** — add an entry to the `class Topic(Enum)`. The **member name** is the topic name in `SCREAMING_SNAKE_CASE` (hyphens → underscores, uppercased); the **value** is the kebab-case topic name. Place it next to its sibling topics (the enum is grouped by family, not strictly alphabetical): ```python - TOPIC_NAME = "" + # e.g. topic "taskworker-launchpad-push": + TASKWORKER_LAUNCHPAD_PUSH = "taskworker-launchpad-push" ``` -2. **`src/sentry/conf/server.py`** — add an entry to the `KAFKA_TOPIC_TO_CLUSTER` dict with value `"default"`, next to the sibling entries: +2. **`src/sentry/conf/server.py`** — add an entry to the `KAFKA_TOPIC_TO_CLUSTER` dict next to the sibling entries. Use `"default"` (every current entry maps to `"default"`); if a sibling in the same family ever uses a different value, match that instead: ```python "": "default", ``` From e6c3b31b51914b21c48094f0e414fb07390e3366 Mon Sep 17 00:00:00 2001 From: Evan Hicks Date: Thu, 25 Jun 2026 12:02:58 -0400 Subject: [PATCH 7/9] feat: support public vs private topics in create-kafka-topic Add a public/private decision up front: - Public (available to self-hosted): unchanged three-PR flow (sentry-kafka-schemas + ops + sentry). - Private (internal regions only): a single ops PR. The per-region overrides carry `cluster:` + `override_topic: ` to inherit the public topic's topic_creation_config. No sentry-kafka-schemas change, no sentry change, no all_topics.yaml entry, and no version bump. Steps 1 (schemas) and 4 (sentry) are marked public-only; Step 0 asks the public/private question and, for private, the override topic; Step 3 and the report handle both paths. SPEC and README updated. Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 2 +- skills/create-kafka-topic/SKILL.md | 84 ++++++++++++++++++++---------- skills/create-kafka-topic/SPEC.md | 44 +++++++++------- 3 files changed, 83 insertions(+), 47 deletions(-) diff --git a/README.md b/README.md index a594a32..abed5b3 100644 --- a/README.md +++ b/README.md @@ -49,7 +49,7 @@ Works with Claude Code, Cursor, Cline, GitHub Copilot, and other compatible agen | [code-simplifier](skills/code-simplifier/SKILL.md) | Simplifies and refines code for clarity, consistency, and maintainability while preserving all functionality. | | [commit](skills/commit/SKILL.md) | ALWAYS use this skill when committing code changes — never commit directly without it. | | [create-branch](skills/create-branch/SKILL.md) | Create git branches following Sentry naming conventions. | -| [create-kafka-topic](skills/create-kafka-topic/SKILL.md) | Create a new Kafka topic across sentry-kafka-schemas, ops, and sentry. | +| [create-kafka-topic](skills/create-kafka-topic/SKILL.md) | Create a new Kafka topic — public (sentry-kafka-schemas + ops + sentry) or private (ops-only, via override_topic). | | [django-access-review](skills/django-access-review/SKILL.md) | Django access control and IDOR security review. | | [django-perf-review](skills/django-perf-review/SKILL.md) | Django performance code review. | | [doc-coauthoring](skills/doc-coauthoring/SKILL.md) | Guide users through a structured workflow for co-authoring documentation. | diff --git a/skills/create-kafka-topic/SKILL.md b/skills/create-kafka-topic/SKILL.md index 5f15749..6472114 100644 --- a/skills/create-kafka-topic/SKILL.md +++ b/skills/create-kafka-topic/SKILL.md @@ -1,30 +1,38 @@ --- name: create-kafka-topic -description: Create a new Kafka topic across the Sentry stack — registers it in sentry-kafka-schemas, ops (shared_config/kafka), and sentry, opening one PR per repo. Use when asked to "create a new Kafka topic", "add a Kafka topic", "register a Kafka topic", "set up a new Kafka topic", or provision a topic in sentry-kafka-schemas/ops/sentry. +description: Create a new Kafka topic for Sentry. Public topics (available to self-hosted) get PRs in sentry-kafka-schemas, ops, and sentry; private topics (internal regions only, inheriting settings from a public topic via override_topic) get a single ops PR. Use when asked to "create a new Kafka topic", "add a Kafka topic", "register a Kafka topic", "set up a new Kafka topic", or provision a public or private Kafka topic. --- # Create a Kafka Topic -Create and register a new Kafka topic across three repos, opening a PR in each: +First decide whether the topic is **public** or **private** (Step 0 asks). The path differs: -1. **sentry-kafka-schemas** — the topic schema definition + CODEOWNERS -2. **ops** — deployment config (default partitions + per-region overrides) -3. **sentry** — the `Topic` enum and cluster mapping +- **Public** — available to self-hosted Sentry. Defined in `sentry-kafka-schemas`, registered in the `sentry` `Topic` enum, and deployed via `ops`. Opens **three PRs**: + 1. **sentry-kafka-schemas** — the topic schema definition + CODEOWNERS + 2. **ops** — deployment config (default partitions + per-region overrides) + 3. **sentry** — the `Topic` enum and cluster mapping +- **Private** — used only in our internal regions; **not** in `sentry-kafka-schemas` or `sentry`. It inherits its `topic_creation_config` from an existing public topic via `override_topic`. Opens **one PR**: just **ops**. -**Requires**: GitHub CLI (`gh`) authenticated, and local checkouts of all three repos. +**Requires**: GitHub CLI (`gh`) authenticated, and local checkouts of the repos you'll touch (all three for public; just `ops` for private). -Run the steps in order. The end deliverable is **three PR links** returned to the user. +Run the steps in order. For **private** topics, do only Step 0, Step 2, Step 3, and Step 5 — **skip Step 1 (schemas) and Step 4 (sentry)**, which are marked *public only*. > **Not getsentry.** getsentry no longer needs a per-topic change — it loads `KAFKA_TOPIC_TO_CLUSTER` at runtime from the topicctl-generated YAML that ops mounts (getsentry/getsentry#20512, #20661). Do not edit `cellsilo.py`. ## Step 0: Gather inputs and locate repos -1. Prompt the user for: +1. **Ask whether the topic is public or private** — this determines the whole path: + - **Public** — available to self-hosted Sentry. Full three-PR flow. + - **Private** — only used in our internal regions. Single ops PR; inherits settings from a public topic via `override_topic`. +2. Prompt the user for: - **Topic name** (kebab-case, e.g. `ingest-foo`) - **Default number of partitions** - - **Owning team** (GitHub team handle, e.g. `@getsentry/taskbroker`) — used for the `sentry-kafka-schemas` CODEOWNERS entry and requested as a **reviewer on all three PRs**. For `gh pr create --reviewer`, pass it as the `org/team` slug **without** the leading `@` (e.g. `getsentry/taskbroker`) — referred to as `` below. -2. Locate each repo in the workspace (`sentry-kafka-schemas`, `ops`, `sentry`). If any is not found near the working directory, ask the user for its path. Use `$SCHEMAS`, `$OPS`, `$SENTRY` to refer to them below. -3. **Check for collision**: if `$SCHEMAS/topics/.yaml` already exists, tell the user the topic already exists and ask for a different name. Do not proceed until the name is free. + - **Owning team** (GitHub team handle, e.g. `@getsentry/taskbroker`) — for **public** topics it is the `sentry-kafka-schemas` CODEOWNERS entry; for **all** topics it is requested as a **reviewer on every PR this skill opens**. For `gh pr create --reviewer`, pass it as the `org/team` slug **without** the leading `@` (e.g. `getsentry/taskbroker`) — referred to as `` below. + - **(private only) Override topic** — the existing **public** topic whose `topic_creation_config` this private topic inherits. It must already exist in `$OPS/shared_config/kafka/topics/defaults/all_topics.yaml` (verify it does); referred to as `` below. +3. Locate the repos you'll touch. **Public**: `sentry-kafka-schemas`, `ops`, `sentry`. **Private**: just `ops`. If a needed repo isn't found near the working directory, ask the user for its path. Use `$SCHEMAS`, `$OPS`, `$SENTRY` to refer to them below. +4. **Check for collision** — ask for a different name and do not proceed until it's free: + - **Public**: `$SCHEMAS/topics/.yaml` must not already exist. + - **Private**: no `$OPS/shared_config/kafka/topics/.yaml` or `$OPS/shared_config/kafka/topics/regional_overrides/*/.yaml` may already exist. Before each repo's work, ensure a clean tree on an updated default branch: @@ -36,7 +44,9 @@ git checkout "$BASE" && git pull --ff-only git checkout -b add-kafka-topic- ``` -## Step 1: sentry-kafka-schemas PR +## Step 1: sentry-kafka-schemas PR — *public only* + +> **Skip this entire step for private topics.** Private topics are not defined in `sentry-kafka-schemas`. Create `$SCHEMAS/topics/.yaml`. Read 2–3 existing files in `$SCHEMAS/topics/` that are closest to this topic's purpose to infer good values (pipeline, producer/consumer services, schema type, resource path). @@ -120,26 +130,35 @@ In `$OPS`: ```yaml disabled: true ``` - - Region enabled, partitions **differ** from the default: - ```yaml - cluster: {cluster_name} - partitions: {region_partitions} - ``` - - Region enabled, partitions **equal** the default → omit `partitions`, write only: - ```yaml - cluster: {cluster_name} - ``` + - Region **enabled** → write `cluster:`; for **private** topics also add `override_topic: `; add `partitions:` only when the region's partitions differ from the default. + - Public, partitions == default → only `cluster: {cluster_name}` + - Public, partitions differ: + ```yaml + cluster: {cluster_name} + partitions: {region_partitions} + ``` + - Private, partitions == default: + ```yaml + cluster: {cluster_name} + override_topic: + ``` + - Private, partitions differ: + ```yaml + cluster: {cluster_name} + override_topic: + partitions: {region_partitions} + ``` 3. **Cookiecutter region template** — also create the regional override in the new-region template so future regions get the topic: `cookiecutters/cookiecutter-region/shared_config/kafka/topics/regional_overrides/{{region}}/.yaml`. New regions start disabled, so its contents are exactly: ```yaml disabled: true ``` (Skipping this is flagged by Warden's `cookiecutter-region-backport` check — see the `taskworker-seer-push.yaml` precedent.) -4. **Register for deployment** — add `` to the `all_deployed_topics:` list in `shared_config/kafka/topics/defaults/all_topics.yaml`. The list is grouped by topic family, not strictly alphabetical — insert it next to its sibling topics (e.g. right after ``). -5. **Bump the schemas dependency** — update the `sentry-kafka-schemas==` pin in `python/requirements.txt` to `` (see the release-dependency note in Step 1). The pin lives only in the compiled `requirements.txt`, not `requirements.in`. This is a clean, complete change (no hashes in this file). +4. **Register for deployment** — *public only*. Add `` to the `all_deployed_topics:` list in `shared_config/kafka/topics/defaults/all_topics.yaml`. The list is grouped by topic family, not strictly alphabetical — insert it next to its sibling topics (e.g. right after ``). **Skip for private topics** — they are intentionally not in `all_topics.yaml`. +5. **Bump the schemas dependency** — *public only*. Update the `sentry-kafka-schemas==` pin in `python/requirements.txt` to `` (see the release-dependency note in Step 1). The pin lives only in the compiled `requirements.txt`, not `requirements.in`. **Skip for private topics** — they reuse an already-released public topic, so no new schemas release is involved. > **Do not edit generated/materialized files.** CI regenerates `shared_config/_materialized_configs/`, `k8s/clusters/*/_topicctl_generated.yaml`, `k8s/materialized_manifests/`, and topicctl job manifests via `make materialize`. Only edit the source files above. -Commit and open the PR: +Commit and open the PR (drop `python/requirements.txt` from the `git add` for private topics): ```bash git add shared_config/kafka/topics/ cookiecutters/cookiecutter-region/ python/requirements.txt @@ -149,7 +168,9 @@ gh pr create --fill --reviewer --title "feat: deploy **Skip this entire step for private topics.** Private topics are not added to the `sentry` `Topic` enum or `KAFKA_TOPIC_TO_CLUSTER`. In `$SENTRY`: @@ -176,9 +197,18 @@ Capture the PR URL. ## Step 5: Report -Return all three PR links to the user **and the dependency ordering between them**, so the user knows what must merge first. The schemas PR must merge and publish a release before the dependency-bump PRs can reference the released version. +Return the PR link(s) to the user. + +**Private topic** — one PR, no cross-PR dependency: + +``` +Opened 1 PR for the private `` topic: + +1. ops: + └─ Inherits topic_creation_config from public topic ``. +``` -Output in this shape (annotate each PR with its dependency): +**Public topic** — three PRs, **with the dependency ordering** so the user knows what must merge first. The schemas PR must merge and publish a release before the dependency-bump PRs can reference the released version: ``` Opened 3 PRs for the `` topic: diff --git a/skills/create-kafka-topic/SPEC.md b/skills/create-kafka-topic/SPEC.md index f5809fd..27e0cdd 100644 --- a/skills/create-kafka-topic/SPEC.md +++ b/skills/create-kafka-topic/SPEC.md @@ -2,21 +2,24 @@ ## Intent -The `create-kafka-topic` skill provisions a brand-new Kafka topic across the three repos that must all agree for a Sentry topic to exist and deploy: `sentry-kafka-schemas` (schema + ownership), `ops` (deployment/partition/region config), and `sentry` (topic enum + cluster mapping). +The `create-kafka-topic` skill provisions a brand-new Kafka topic. There are two paths, chosen up front: -`getsentry` is intentionally **not** in scope: it no longer maintains a per-topic cluster map by hand — it loads `KAFKA_TOPIC_TO_CLUSTER` at runtime from the topicctl-generated YAML mounted by ops (getsentry/getsentry#20512, #20661). +- **Public** (available to self-hosted): spans the three repos that must agree for a public topic to exist and deploy — `sentry-kafka-schemas` (schema + ownership), `ops` (deployment/partition/region config), and `sentry` (topic enum + cluster mapping). Three PRs. +- **Private** (internal regions only): defined solely in `ops`, inheriting `topic_creation_config` from an existing public topic via `override_topic`. It is not in `sentry-kafka-schemas`, the `sentry` `Topic` enum, or `all_topics.yaml`. One PR (ops only). -Its main job is to turn a topic name and a few deployment choices into one correct, reviewable PR per repo, matching the conventions of the topics that already exist, and to hand the user the resulting PR links. +`getsentry` is intentionally **not** in scope for either path: it no longer maintains a per-topic cluster map by hand — it loads `KAFKA_TOPIC_TO_CLUSTER` at runtime from the topicctl-generated YAML mounted by ops (getsentry/getsentry#20512, #20661). + +Its main job is to turn a topic name and a few deployment choices into one correct, reviewable PR per touched repo, matching the conventions of the topics that already exist, and to hand the user the resulting PR links. ## Scope In scope: +- Determining whether the topic is public or private, and for private topics the `override_topic` (an existing public topic). - Prompting for the topic name, default partition count, owning team, and per-region enablement/partitions/cluster. -- Creating the schema file and CODEOWNERS entry in `sentry-kafka-schemas`. -- Creating the default partition file, per-region override files, the cookiecutter-region template override (`disabled: true`), and `all_topics.yaml` entry in `ops`, plus the `sentry-kafka-schemas` requirement bump. -- Adding the `Topic` enum entry and `KAFKA_TOPIC_TO_CLUSTER` entry in `sentry`, plus the `sentry-kafka-schemas` dependency bump. -- Opening one PR per repo and reporting all three URLs. +- **Public:** schema file + CODEOWNERS entry in `sentry-kafka-schemas`; default partition file, per-region overrides, cookiecutter-region template override, `all_topics.yaml` entry, and `sentry-kafka-schemas` requirement bump in `ops`; `Topic` enum + `KAFKA_TOPIC_TO_CLUSTER` entry + dependency bump in `sentry`. Three PRs. +- **Private:** in `ops` only — per-region overrides carrying `cluster:` + `override_topic:` (+ optional `partitions`), default partition file, and cookiecutter-region template override. No `all_topics.yaml` entry, no schemas/sentry changes, no version bump. One PR. +- Opening one PR per touched repo, each with the owning team as reviewer, and reporting the URLs. Out of scope: @@ -29,35 +32,37 @@ Out of scope: ## Users And Trigger Context - Primary users: Sentry engineers (and coding agents) standing up a new Kafka topic. -- Common requests: "create a new Kafka topic", "add a Kafka topic", "register a Kafka topic", "set up a new Kafka topic". +- Common requests: "create a new Kafka topic", "add a Kafka topic", "register a Kafka topic", "set up a new Kafka topic" (public or private). - Should not trigger for: changing an existing topic's partitions/clusters, debugging consumers, or general Kafka questions. ## Runtime Contract -- Required first actions: prompt for topic name, default partitions, and owning team; locate all three repos; verify the topic name is not already taken in `sentry-kafka-schemas/topics/`. +- Required first actions: determine public vs private; prompt for topic name, default partitions, and owning team (and, for private, the `override_topic`); locate the repos for the chosen path; verify the name is free (public: `sentry-kafka-schemas/topics/`; private: `ops` default + regional override paths). - Per-repo precondition: a clean working tree on an updated default branch before creating the topic branch. -- Required outputs: three pull requests (one per repo), each with the owning team requested as a reviewer, plus the three URLs returned to the user. +- Required outputs: public → three PRs (one per repo); private → one ops PR. Each PR has the owning team requested as a reviewer; return the URL(s). - Non-negotiable constraints: - Always ask the user which regions to enable; never infer the enabled set from a reference topic, even when told to "use the same values as ``". - Do not edit generated/materialized files in `ops`, and do not edit `getsentry`. - - Per-region override contents follow the exact rules: `disabled: true` when not enabled; `cluster:` only when enabled and partitions equal the default; `cluster:` + `partitions:` when they differ. + - Per-region override contents: `disabled: true` when not enabled. When enabled, `cluster:` (+ `partitions:` only if it differs from the default); **private topics also include `override_topic: `**. + - Private topics: ops PR only — no `sentry-kafka-schemas` change, no `sentry` change, no `all_topics.yaml` entry, no version bump. The `override_topic` must be an existing public topic in `all_topics.yaml`. - Insert enum / `all_topics.yaml` entries next to sibling topics (grouped by family), not alphabetically. - - Confirm the generated schema file with the user before opening the first PR. -- Sequencing: the schemas PR gates the dependency bumps in `sentry`/`ops`. The ops/sentry PRs still bump the pin to the anticipated next version (latest released patch + 1) so the required change is visible; the PR body must flag it as anticipated and, for sentry, note that `uv.lock` must be regenerated with `uv lock` once the release publishes. + - Confirm the generated schema file with the user before opening the first PR (public). +- Sequencing (public only): the schemas PR gates the dependency bumps in `sentry`/`ops`. The ops/sentry PRs still bump the pin to the anticipated next version (latest released patch + 1) so the required change is visible; the PR body must flag it as anticipated and, for sentry, note that `uv.lock` must be regenerated with `uv lock` once the release publishes. Private topics have no such dependency. - Expected bundled files loaded at runtime: only `SKILL.md`. ## Evaluation A run is correct when: -- All three PRs are opened, each containing only the source-of-truth changes for its repo (no generated files). -- The schema YAML, region overrides, enum entry, and cluster mappings match the conventions of comparable existing topics. +- The right PRs are opened for the path (public → three; private → one ops PR), each containing only the source-of-truth changes for its repo (no generated files). +- The schema YAML, region overrides, enum entry, and cluster mappings match the conventions of comparable existing topics; private overrides carry `override_topic` pointing at an existing public topic. - The enabled-region set matches what the user explicitly chose (not a reference topic's set). -- The user receives three working PR links. +- The user receives the correct PR link(s). Reference examples: -- `taskworker-ingest-push` — getsentry/sentry-kafka-schemas#484, getsentry/ops#20917, getsentry/sentry#117135 (its getsentry/getsentry#20495 predates the loader change and is no longer needed). -- `taskworker-launchpad-push` — getsentry/sentry-kafka-schemas#490, getsentry/ops#21361, getsentry/sentry#118161 (s4s2-only enablement, schemas-release-deferred version bumps). +- Public: `taskworker-ingest-push` — getsentry/sentry-kafka-schemas#484, getsentry/ops#20917, getsentry/sentry#117135 (its getsentry/getsentry#20495 predates the loader change and is no longer needed). +- Public: `taskworker-launchpad-push` — getsentry/sentry-kafka-schemas#490, getsentry/ops#21361, getsentry/sentry#118161 (s4s2-only enablement, schemas-release-deferred version bumps). +- Private (override_topic): `ops` `regional_overrides/us/taskworker-subscriptions-dlq.yaml` (`cluster: kafka-small` + `override_topic: taskworker-dlq`), not present in `all_topics.yaml` or `sentry`. ## Limitations @@ -67,6 +72,7 @@ Reference examples: ## Maintenance -- If the three-repo wiring changes (new repo, renamed config dict, new required file, or getsentry again needing a per-topic edit), update the corresponding step and this spec. +- If the public three-repo wiring changes (new repo, renamed config dict, new required file, or getsentry again needing a per-topic edit), update the corresponding step and this spec. +- If the private-topic mechanism changes (e.g. `override_topic` semantics in `topic_builder.libsonnet`, or private topics start requiring schema/sentry entries), update Steps 0/3 and this spec. - If `ops` changes how regions or clusters are configured, update Steps 2–3. - Keep the reference example current if the canonical topic-creation PR set changes. From aefd099b7935879171a0d2348bdb3811b6c1c6d8 Mon Sep 17 00:00:00 2001 From: Evan Hicks Date: Fri, 26 Jun 2026 10:08:49 -0400 Subject: [PATCH 8/9] fix: address PR review feedback on create-kafka-topic - Collect a reference/sibling topic in Step 0 (it was used by Steps 2-4 but never gathered). - Broaden the collision check to all locations (sentry-kafka-schemas, ops default + regional overrides, and the sentry Topic enum / cluster map), regardless of public/private (was schemas-only for public). - Add a clarifying question + caveat that not every public topic needs a new schema entry; outcomes topics in particular reuse an existing schema and must not be added to sentry-kafka-schemas (per reviewer feedback). Co-Authored-By: Claude Opus 4.8 (1M context) --- skills/create-kafka-topic/SKILL.md | 18 +++++++++++++----- skills/create-kafka-topic/SPEC.md | 2 +- 2 files changed, 14 insertions(+), 6 deletions(-) diff --git a/skills/create-kafka-topic/SKILL.md b/skills/create-kafka-topic/SKILL.md index 6472114..c19595e 100644 --- a/skills/create-kafka-topic/SKILL.md +++ b/skills/create-kafka-topic/SKILL.md @@ -22,17 +22,23 @@ Run the steps in order. For **private** topics, do only Step 0, Step 2, Step 3, ## Step 0: Gather inputs and locate repos 1. **Ask whether the topic is public or private** — this determines the whole path: - - **Public** — available to self-hosted Sentry. Full three-PR flow. + - **Public** — available to self-hosted Sentry. Up to three PRs. - **Private** — only used in our internal regions. Single ops PR; inherits settings from a public topic via `override_topic`. 2. Prompt the user for: - **Topic name** (kebab-case, e.g. `ingest-foo`) - **Default number of partitions** + - **Reference topic** — the closest existing topic of the same family. It drives region discovery (Step 2) and where the `all_topics.yaml` / `Topic` enum entries go (Steps 3–4). Referred to as `` below. - **Owning team** (GitHub team handle, e.g. `@getsentry/taskbroker`) — for **public** topics it is the `sentry-kafka-schemas` CODEOWNERS entry; for **all** topics it is requested as a **reviewer on every PR this skill opens**. For `gh pr create --reviewer`, pass it as the `org/team` slug **without** the leading `@` (e.g. `getsentry/taskbroker`) — referred to as `` below. - **(private only) Override topic** — the existing **public** topic whose `topic_creation_config` this private topic inherits. It must already exist in `$OPS/shared_config/kafka/topics/defaults/all_topics.yaml` (verify it does); referred to as `` below. -3. Locate the repos you'll touch. **Public**: `sentry-kafka-schemas`, `ops`, `sentry`. **Private**: just `ops`. If a needed repo isn't found near the working directory, ask the user for its path. Use `$SCHEMAS`, `$OPS`, `$SENTRY` to refer to them below. -4. **Check for collision** — ask for a different name and do not proceed until it's free: - - **Public**: `$SCHEMAS/topics/.yaml` must not already exist. - - **Private**: no `$OPS/shared_config/kafka/topics/.yaml` or `$OPS/shared_config/kafka/topics/regional_overrides/*/.yaml` may already exist. +3. **(public only) Confirm a new schema definition is actually needed.** Not every public topic adds a new `sentry-kafka-schemas` entry — ask the user: + - **taskworker** topics do get their own `topics/.yaml` (reusing the shared `TaskActivation` protobuf) → proceed with Step 1. + - **outcomes** topics are an exception: they reuse an existing schema and infra-specific variants should **not** be added to `sentry-kafka-schemas`. For an outcomes-style topic, skip Step 1 — confirm the exact handling with the topic owner. + - If the topic reuses an existing public topic's settings wholesale and adds nothing new to the schema registry, it is really a **private** topic — use `override_topic` (Step 3 only). +4. Locate the repos you'll touch. **Public**: `sentry-kafka-schemas`, `ops`, `sentry`. **Private**: just `ops`. If a needed repo isn't found near the working directory, ask the user for its path. Use `$SCHEMAS`, `$OPS`, `$SENTRY` to refer to them below. +5. **Check for collision across every location** (regardless of public/private) — ask for a different name and do not proceed until it's free: + - `$SCHEMAS/topics/.yaml` + - `$OPS/shared_config/kafka/topics/.yaml` and `$OPS/shared_config/kafka/topics/regional_overrides/*/.yaml` + - `$SENTRY` `Topic` enum / `KAFKA_TOPIC_TO_CLUSTER` (grep the kebab name in `src/sentry/conf/types/kafka_definition.py` and `src/sentry/conf/server.py`) Before each repo's work, ensure a clean tree on an updated default branch: @@ -47,6 +53,8 @@ git checkout -b add-kafka-topic- ## Step 1: sentry-kafka-schemas PR — *public only* > **Skip this entire step for private topics.** Private topics are not defined in `sentry-kafka-schemas`. +> +> **Caveat — not every public topic needs a schema entry.** Confirm (Step 0.3) that this topic actually introduces a new schema. **outcomes** topics in particular reuse an existing schema and must not be added to `sentry-kafka-schemas` (infra-specific variants don't belong in the registry that self-hosted consumes). When in doubt, ask the topic owner before creating the file. Create `$SCHEMAS/topics/.yaml`. Read 2–3 existing files in `$SCHEMAS/topics/` that are closest to this topic's purpose to infer good values (pipeline, producer/consumer services, schema type, resource path). diff --git a/skills/create-kafka-topic/SPEC.md b/skills/create-kafka-topic/SPEC.md index 27e0cdd..85c00e6 100644 --- a/skills/create-kafka-topic/SPEC.md +++ b/skills/create-kafka-topic/SPEC.md @@ -37,7 +37,7 @@ Out of scope: ## Runtime Contract -- Required first actions: determine public vs private; prompt for topic name, default partitions, and owning team (and, for private, the `override_topic`); locate the repos for the chosen path; verify the name is free (public: `sentry-kafka-schemas/topics/`; private: `ops` default + regional override paths). +- Required first actions: determine public vs private; prompt for topic name, default partitions, a reference/sibling topic, and owning team (and, for private, the `override_topic`); for public, confirm a new schema is actually needed (outcomes topics are an exception); locate the repos for the chosen path; verify the name is free across **all** locations (`sentry-kafka-schemas/topics/`, `ops` default + regional override paths, and the `sentry` `Topic` enum / `KAFKA_TOPIC_TO_CLUSTER`). - Per-repo precondition: a clean working tree on an updated default branch before creating the topic branch. - Required outputs: public → three PRs (one per repo); private → one ops PR. Each PR has the owning team requested as a reviewer; return the URL(s). - Non-negotiable constraints: From 26adc81aadae04089b347c4fb46c72bc7d096ad1 Mon Sep 17 00:00:00 2001 From: Evan Hicks Date: Fri, 26 Jun 2026 10:43:06 -0400 Subject: [PATCH 9/9] fix: reconcile no-schema public path and private collision check Two inconsistencies introduced by the public/private split: - Outcomes-style public topics skip Step 1 (no new schema), but the version bumps (Steps 3/4) and the report (Step 5) still assumed a schemas release. Gate the bumps and add a two-PR report variant (ops + sentry, no bump) for the reuse-existing-schema case. - The collision check referenced $SCHEMAS/$SENTRY for private topics, which only locate $OPS. Scope the collision check to the repos each path locates (ops always; schemas + sentry for public). Co-Authored-By: Claude Opus 4.8 (1M context) --- skills/create-kafka-topic/SKILL.md | 24 ++++++++++++++++-------- skills/create-kafka-topic/SPEC.md | 4 ++-- 2 files changed, 18 insertions(+), 10 deletions(-) diff --git a/skills/create-kafka-topic/SKILL.md b/skills/create-kafka-topic/SKILL.md index c19595e..07a189c 100644 --- a/skills/create-kafka-topic/SKILL.md +++ b/skills/create-kafka-topic/SKILL.md @@ -35,10 +35,9 @@ Run the steps in order. For **private** topics, do only Step 0, Step 2, Step 3, - **outcomes** topics are an exception: they reuse an existing schema and infra-specific variants should **not** be added to `sentry-kafka-schemas`. For an outcomes-style topic, skip Step 1 — confirm the exact handling with the topic owner. - If the topic reuses an existing public topic's settings wholesale and adds nothing new to the schema registry, it is really a **private** topic — use `override_topic` (Step 3 only). 4. Locate the repos you'll touch. **Public**: `sentry-kafka-schemas`, `ops`, `sentry`. **Private**: just `ops`. If a needed repo isn't found near the working directory, ask the user for its path. Use `$SCHEMAS`, `$OPS`, `$SENTRY` to refer to them below. -5. **Check for collision across every location** (regardless of public/private) — ask for a different name and do not proceed until it's free: - - `$SCHEMAS/topics/.yaml` - - `$OPS/shared_config/kafka/topics/.yaml` and `$OPS/shared_config/kafka/topics/regional_overrides/*/.yaml` - - `$SENTRY` `Topic` enum / `KAFKA_TOPIC_TO_CLUSTER` (grep the kebab name in `src/sentry/conf/types/kafka_definition.py` and `src/sentry/conf/server.py`) +5. **Check for collision in every repo you located** — ask for a different name and do not proceed until it's free: + - `$OPS/shared_config/kafka/topics/.yaml` and `$OPS/shared_config/kafka/topics/regional_overrides/*/.yaml` (always — covers both private topics and the ops files of deployed public topics). + - **Public only** (these repos are located only for public): `$SCHEMAS/topics/.yaml`, and the `$SENTRY` `Topic` enum / `KAFKA_TOPIC_TO_CLUSTER` (grep the kebab name in `src/sentry/conf/types/kafka_definition.py` and `src/sentry/conf/server.py`). Before each repo's work, ensure a clean tree on an updated default branch: @@ -162,7 +161,7 @@ In `$OPS`: ``` (Skipping this is flagged by Warden's `cookiecutter-region-backport` check — see the `taskworker-seer-push.yaml` precedent.) 4. **Register for deployment** — *public only*. Add `` to the `all_deployed_topics:` list in `shared_config/kafka/topics/defaults/all_topics.yaml`. The list is grouped by topic family, not strictly alphabetical — insert it next to its sibling topics (e.g. right after ``). **Skip for private topics** — they are intentionally not in `all_topics.yaml`. -5. **Bump the schemas dependency** — *public only*. Update the `sentry-kafka-schemas==` pin in `python/requirements.txt` to `` (see the release-dependency note in Step 1). The pin lives only in the compiled `requirements.txt`, not `requirements.in`. **Skip for private topics** — they reuse an already-released public topic, so no new schemas release is involved. +5. **Bump the schemas dependency** — *only when Step 1 added a new schema* (i.e. a new `sentry-kafka-schemas` release is required). Update the `sentry-kafka-schemas==` pin in `python/requirements.txt` to `` (see the release-dependency note in Step 1). The pin lives only in the compiled `requirements.txt`, not `requirements.in`. **Skip the bump** for private topics and for public topics that reuse an existing schema (Step 1 skipped) — no new release is involved. > **Do not edit generated/materialized files.** CI regenerates `shared_config/_materialized_configs/`, `k8s/clusters/*/_topicctl_generated.yaml`, `k8s/materialized_manifests/`, and topicctl job manifests via `make materialize`. Only edit the source files above. @@ -191,7 +190,7 @@ In `$SENTRY`: ```python "": "default", ``` -3. **Bump the schemas dependency** — update `sentry-kafka-schemas>=` in `pyproject.toml` to ``, and bump the matching `specifier = ">=..."` line for `sentry-kafka-schemas` in `uv.lock` so the two agree. If `` is already published, run `uv lock` to fully regenerate. If it is **not** published yet (the usual case when this PR precedes the schemas release), you cannot regenerate `uv.lock`'s resolved version + wheel hash — leave the `[[package]]` `version`/`wheels` block as-is and note in the PR that `uv lock` must be re-run once the release publishes. See the release-dependency note in Step 1. +3. **Bump the schemas dependency** — *only when Step 1 added a new schema* (skip when the topic reuses an existing schema, e.g. outcomes — no new release exists to pin to). Update `sentry-kafka-schemas>=` in `pyproject.toml` to ``, and bump the matching `specifier = ">=..."` line for `sentry-kafka-schemas` in `uv.lock` so the two agree. If `` is already published, run `uv lock` to fully regenerate. If it is **not** published yet (the usual case when this PR precedes the schemas release), you cannot regenerate `uv.lock`'s resolved version + wheel hash — leave the `[[package]]` `version`/`wheels` block as-is and note in the PR that `uv lock` must be re-run once the release publishes. See the release-dependency note in Step 1. Commit and open the PR: @@ -216,7 +215,7 @@ Opened 1 PR for the private `` topic: └─ Inherits topic_creation_config from public topic ``. ``` -**Public topic** — three PRs, **with the dependency ordering** so the user knows what must merge first. The schemas PR must merge and publish a release before the dependency-bump PRs can reference the released version: +**Public topic with a new schema** (Step 1 produced a PR) — three PRs, **with the dependency ordering** so the user knows what must merge first. The schemas PR must merge and publish a release before the dependency-bump PRs can reference the released version: ``` Opened 3 PRs for the `` topic: @@ -233,7 +232,16 @@ Merge order: #1 (and its release) → then #2 and #3. If a dependency bump was deferred because the schemas release was not yet published, say so explicitly and note which PR(s) still need the version updated. +**Public topic that reuses an existing schema** (Step 1 skipped, e.g. outcomes) — two PRs, **no schemas release and no version bump**, so no cross-PR ordering: + +``` +Opened 2 PRs for the `` topic: + +1. ops: +2. sentry: +``` + ## Notes - For commit messages and PR descriptions, you may use the `commit` and `pr-writer` skills to follow Sentry conventions; the commands above are a self-contained fallback. -- The schemas PR (Step 1) gates the dependency bumps in ops (Step 3) and sentry (Step 4). Bump the pins to the anticipated `` anyway so reviewers see the required change; flag in each PR body that the version is anticipated (confirm against the actual release) and, for sentry, that `uv.lock` needs `uv lock` once the release publishes. CI on those PRs may stay red until then. +- When Step 1 adds a new schema, its PR gates the dependency bumps in ops (Step 3) and sentry (Step 4). Bump the pins to the anticipated `` anyway so reviewers see the required change; flag in each PR body that the version is anticipated (confirm against the actual release) and, for sentry, that `uv.lock` needs `uv lock` once the release publishes. CI on those PRs may stay red until then. When Step 1 is skipped (private, or public reusing an existing schema), there is no release dependency and no version bump. diff --git a/skills/create-kafka-topic/SPEC.md b/skills/create-kafka-topic/SPEC.md index 85c00e6..eab8c19 100644 --- a/skills/create-kafka-topic/SPEC.md +++ b/skills/create-kafka-topic/SPEC.md @@ -39,7 +39,7 @@ Out of scope: - Required first actions: determine public vs private; prompt for topic name, default partitions, a reference/sibling topic, and owning team (and, for private, the `override_topic`); for public, confirm a new schema is actually needed (outcomes topics are an exception); locate the repos for the chosen path; verify the name is free across **all** locations (`sentry-kafka-schemas/topics/`, `ops` default + regional override paths, and the `sentry` `Topic` enum / `KAFKA_TOPIC_TO_CLUSTER`). - Per-repo precondition: a clean working tree on an updated default branch before creating the topic branch. -- Required outputs: public → three PRs (one per repo); private → one ops PR. Each PR has the owning team requested as a reviewer; return the URL(s). +- Required outputs: public with a new schema → three PRs; public reusing an existing schema (Step 1 skipped, e.g. outcomes) → two PRs (ops + sentry), no version bump; private → one ops PR. Each PR has the owning team requested as a reviewer; return the URL(s). - Non-negotiable constraints: - Always ask the user which regions to enable; never infer the enabled set from a reference topic, even when told to "use the same values as ``". - Do not edit generated/materialized files in `ops`, and do not edit `getsentry`. @@ -47,7 +47,7 @@ Out of scope: - Private topics: ops PR only — no `sentry-kafka-schemas` change, no `sentry` change, no `all_topics.yaml` entry, no version bump. The `override_topic` must be an existing public topic in `all_topics.yaml`. - Insert enum / `all_topics.yaml` entries next to sibling topics (grouped by family), not alphabetically. - Confirm the generated schema file with the user before opening the first PR (public). -- Sequencing (public only): the schemas PR gates the dependency bumps in `sentry`/`ops`. The ops/sentry PRs still bump the pin to the anticipated next version (latest released patch + 1) so the required change is visible; the PR body must flag it as anticipated and, for sentry, note that `uv.lock` must be regenerated with `uv lock` once the release publishes. Private topics have no such dependency. +- Sequencing (only when Step 1 adds a new schema): the schemas PR gates the dependency bumps in `sentry`/`ops`. When the topic reuses an existing schema (outcomes) or is private, there is no schemas release, no version bump, and no cross-PR ordering. The ops/sentry PRs still bump the pin to the anticipated next version (latest released patch + 1) so the required change is visible; the PR body must flag it as anticipated and, for sentry, note that `uv.lock` must be regenerated with `uv lock` once the release publishes. Private topics have no such dependency. - Expected bundled files loaded at runtime: only `SKILL.md`. ## Evaluation