refactor(cell/infra): consume the framework's celld-fleet module

[coccyx]

Replaces this stack's hand-rolled fleet infra with the framework's infra/celld-fleet module (extracted from here in PRs #129/#130/#132), keeping the live stack's identity byte-for-byte.

Applied to the live cell already — plan read line by line, applied in a quiet window, verified end-to-end. Details below.

Depends on framework PR criblio/cribl-search-app-framework#34 (adds security_group_description). That must merge, and .framework-sha must move to include it, before this can plan on another machine — see "Before merging".

What's here

cell/infra/main.tf is now a module block plus moved blocks. This root keeps only its identity and its AWS session:

Input	Value
cell_name	apm-cell → every resource stays apm-cell-node
bucket_name	cribl-apm-cell-test
ssm_parameter_prefix	/apm-cell
secret_env_keys	WEBHOOK_BEARER, UI_BEARER, TICKET_SECRET, LLM_API_KEY, CRIBL_CLIENT_ID, CRIBL_CLIENT_SECRET
plain_env	LLM_BASE_URL, LLM_MODEL, CRIBL_BASE_URL, CRIBL_DATASET

user_data.sh.tftpl is deleted (the module owns it). The backend block and provider stay in this root unchanged. Outputs are re-exported so terraform output -raw cell_url and instance_id keep working.

Plan summary

12 moved blocks re-address every resource into module.fleet.*. Final plan:

Plan: 2 to add, 0 to change, 2 to destroy.

Moved with no replacement (10): the bucket + its 3 config resources, the IAM role / policy / attachment / instance profile, the security group, the EIP.

Replaced (2): aws_instance.cell and aws_eip_association.cell (the association follows the instance id).

Three forces replacement lines, and it's worth being precise about why, because only one of them is this PR's doing:

1. user_data — the generalized template differs textually under user_data_replace_on_change = true. This is the designed adoption path.
2. ami — the AL2023 data.aws_ssm_parameter tracks "latest" and the image rolled. Pre-existing drift, not caused by this refactor.
3. instance_id on the association — a consequence of (1)/(2).

I confirmed (2) by planning origin/master's unmodified config against live state first: it also reported 2 to add, 0 to change, 2 to destroy and also wanted the instance replaced. So the refactor itself contributes zero additional churn — same plan shape before and after.

The security group did not churn

description is a create-time argument on aws_security_group, and the group's name derives from cell_name, so both configs resolve to apm-cell-node — a same-name destroy/create that fights the attached instance. The module's generic wording would have triggered exactly that. Counterfactual, dropping the pin:

# module.fleet.aws_security_group.cell must be replaced
  ~ description = "Investigator cell: HTTPS in (Cribl webhooks + platform proxy), …"
              -> "celld cell node: HTTPS in (webhooks + platform proxy), …" # forces replacement
Plan: 3 to add, 0 to change, 3 to destroy.

Hence framework PR #34, which makes it an input so this root pins its original text. With the pin the SG is a pure move.

user_data is functionally identical

Rendered both templates with the live values and diffed: same six secret keys, same four CELLD_VAR_* plain vars, same /var/log/apm-cell-init.log, both pass bash -n. The systemd-unit heredoc changed from quoted to unquoted, but the body contains no $, so nothing expands unexpectedly. The diff is the generalization itself, nothing behavioral.

One behavior change

The old template pulled the LLM/Cribl secrets only when llm_base_url was non-empty — a stub-agent fallback. The module reads secret_env_keys unconditionally, so boot now fails loudly on any missing parameter. All six exist in the test account (verified SecureString, versions 1–2), so this is a no-op here. But "clear llm_base_url to get stub mode" no longer works; shorten secret_env_keys instead. Documented in cell/infra/README.md.

Verification on the live cell

Applied at ~16:08 UTC with nothing running or queued (active (running/queued): 0).

Apply complete! Resources: 2 added, 0 changed, 2 destroyed.
cell_url    = "https://54-71-34-177.sslip.io"   # unchanged
public_ip   = "54.71.34.177"                    # unchanged (EIP not replaced)
instance_id = "i-08a186846843a8d63"             # was i-0ee35cbacf3c7e237

• /healthz → {"ok":true,"disabled":false}, HTTP 200 within 15s.
• cloud-init status → done; celld and caddy both active; celld 0.2.0, Caddy 2.10.2.
• journalctl -u celld -p warning → -- No entries --. All InvestigationDO replicas restored from the bucket (restored remote replica … fresh=false), so durable state survived the replacement.
• Env file → all 13 vars present, including the six CELLD_VAR_* secrets.
• One investigation end-to-end — fired a synthetic frontend-proxy alert, concluded in ~2min, 47 frames: real KQL + PromQL against staging, correct root cause (product-catalog returning Product Not Found for garbage product IDs, cascading via the frontend BFF), concluded through present_investigation_summary.
• Dataset commits → started + investigated rows for inv-58d0afb1… present in otel (record_kind=="investigation").

A payment alert I fired first hit the 12-turn cap and failed. That's pre-existing agent behavior, not infra: payment failed the same way at 05:27 today and a frontend-proxy run failed at 15:52, both before this apply. Its started / investigation_failed rows committed correctly, so the lifecycle path works. Worth a separate look at turn budget for services whose alert window has already gone quiet.

Before merging

main.tf points at a sibling checkout (../../../cribl-search-app-framework/infra/celld-fleet), matching how the app consumes the framework's npm packages. Two consequences:

1. Framework PR feat(service-detail): section-grouped layout + always-render metric cards #34 must merge first, and .framework-sha needs to move to a SHA that includes both the module and feat(service-detail): section-grouped layout + always-render metric cards #34 — the currently pinned 1c37201 predates infra/ entirely, so a fresh clone can't terraform init this. (I planned against a worktree of feat(service-detail): section-grouped layout + always-render metric cards #34's branch; the committed path is what CI/other machines will use.)
2. A divergent local framework checkout silently changes the plan. Worth deciding whether this should become a pinned git::…?ref=<tag> source instead — the module README suggests it for CI. I left the sibling path to match repo convention, but I'd lean toward pinning it; happy to follow up.

Test plan

cd cell/infra
terraform init          # resolves the module from the sibling framework checkout
terraform plan -var bucket_name=cribl-apm-cell-test \
  -var llm_base_url=https://openrouter.ai/api/v1 \
  -var llm_model=deepseek/deepseek-v4-flash-0731 \
  -var cribl_base_url=https://main-objective-shirley-sho21r7.cribl-staging.cloud \
  -var cribl_dataset=otel
# expect: No changes. (post-apply)
curl -s https://54-71-34-177.sslip.io/healthz

🤖 Generated with Claude Code