Skip to content

Add log buffer and reduce info log noise - #814

Merged
stewartshea merged 10 commits into
mainfrom
logging-refactor-structured-logs-ui
Jul 31, 2026
Merged

Add log buffer and reduce info log noise#814
stewartshea merged 10 commits into
mainfrom
logging-refactor-structured-logs-ui

Conversation

@stewartshea

Copy link
Copy Markdown
Contributor
  • Introduce a thread-safe ring buffer to store recent log entries
  • Capture rendering warnings/errors in the buffer for UI display
  • Add a "Logs" tab to the explorer and a "Recent Issues" card to the home page
  • Add phase durations and recent errors to the health endpoint
  • Switch frequent discovery and resolution logs from info to debug level
  • Add per-module log level configuration via environment variables

stewartshea and others added 9 commits July 31, 2026 10:30
- Introduce a thread-safe ring buffer to store recent log entries
- Capture rendering warnings/errors in the buffer for UI display
- Add a "Logs" tab to the explorer and a "Recent Issues" card to the
  home page
- Add phase durations and recent errors to the health endpoint
- Switch frequent discovery and resolution logs from info to debug level
- Add per-module log level configuration via environment variables
Introduce JSON console logging as the default format
(`LOG_FORMAT=json`),
per-module level control via `LOG_LEVEL_INDEXERS`,
`LOG_LEVEL_ENRICHERS`,
`LOG_LEVEL_RENDERERS`, and `LOG_LEVEL_WORKSPACE_BUILDER`, a new Logs tab
in the Workspace Explorer with live filtering, a `/explorer/api/logs`
endpoint for programmatic access, and an enhanced `/health/` endpoint
reporting phase durations and recent errors. Update documentation
throughout to describe the new logging features, the UI Logs tab, and
the workbench builder's page updates.
The CLI now uses StructuredJsonFormatter for machine-readable output,
consistent with the server. The explorer page renders log entries with
a structured layout and corrects field references from `kind` to
`artifact_kind` to match the updated data model.
These are expected conditions (e.g., missing configuration, lookup
failures) that don't warrant warnings. Also fixes a regex in a GCP
test to properly match the backend selection message.
Also fix a test assertion in the GCP/K8s test to use the correct
log file and simplify the regex pattern. Update the home dashboard
to fetch the warning count from the API instead of relying on the
last run's stored value.

@stewartshea stewartshea left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Structured Logging Refactor — Review

Branch: logging-refactor-structured-logs-ui
34 files, +991 −230

What changed

New modules (2 files):

  • src/workspace_builder/log_buffer.py — Thread-safe ring buffer (500 entries) + FileLogSink that persists full log history to ${RW_LOG_FILE:-/tmp/runwhen-logs.jsonl}
  • src/workspace_builder/log_formatter.pyStructuredJsonFormatter: {"timestamp","level","message","logger","module"}

Structured JSON logging (default on):

  • src/workspace_builder/startup.pyLOG_FORMAT=json is the default console format. Per-module log levels via LOG_LEVEL_INDEXERS, LOG_LEVEL_ENRICHERS, LOG_LEVEL_RENDERERS, LOG_LEVEL_WORKSPACE_BUILDER. DEBUG_LOGGING=true still works as shortcut. Uvicorn loggers also configured for JSON. Ring buffer + file sink always active.
  • src/run.py — CLI client now wires StructuredJsonFormatter so its output matches the server process
  • src/config_reloader.py — Kubeconfig reloader uses JSON format
  • src/azure_utils.py, src/k8s_utils.pyprint()logger.info/warning/error

Log API + Web UI:

  • GET /explorer/api/logs?level=&phase=&limit= — Query the ring buffer
  • GET /explorer/api/logs/download — Download full JSONL history
  • /health/ — Now includes phase_durations + recent_errors
  • explorer.html — New "Logs" tab (2nd tab) with level/phase filters, auto-refresh, terminal-style compact rows, template error highlighting
  • home.html — "Recent Issues" card with real ERROR content, warnings tile counts from ring buffer (matches Logs tab), clickable → pre-filtered Logs tab

Log level hygiene (~90 demotions):

  • 51 logger.warninglogger.info (expected fallbacks/skipping/graceful conditions)
  • 15 logger.warninglogger.debug (diagnostic detail, SLX name shortening, qualifier resolution)
  • 1 logger.infologger.debug (per-item rendering)
  • Remaining WARNING calls are genuine problems (auth failures, git errors, connection issues)

"All Kinds" fix:

  • explorer.html sidebar used wrong key k.kind → fixed to k.artifact_kind (5 locations)

Tests:

  • .test/gcp/gcp-and-k8s/Taskfile.yaml — Grep pattern updated for JSON log format

Docs (4 files):

  • docs/user-guide/troubleshooting/stuck.md — Web UI Logs Tab, Logs API, Structured JSON Logging sections
  • docs/user-guide/troubleshooting/cloudquery-debug-logging.md — Expanded to full logging configuration reference
  • docs/user-guide/features/workspace-builder.md — Logs tab + Recent Issues documentation
  • README.md — Logs tab + structured logging bullets

How to use

# Structured JSON is the DEFAULT — no flags needed
./run.sh

# Per-module debug (only indexers, everything else stays at INFO)
LOG_LEVEL_INDEXERS=DEBUG ./run.sh

# Plain-text format (opt-out)
LOG_FORMAT=simple ./run.sh

# Download full log history
curl http://localhost:8000/explorer/api/logs/download > runwhen-logs.jsonl

# Query recent errors
curl "http://localhost:8000/explorer/api/logs?level=ERROR&limit=10"

Verify

# Logs should be valid JSON
curl -s http://localhost:8000/explorer/api/logs?limit=1 | jq .entries[0].timestamp

# Health should include timing
curl -s http://localhost:8000/health/ | jq .phase_durations

@stewartshea

Copy link
Copy Markdown
Contributor Author

Structured Logging Refactor

Branch: logging-refactor-structured-logs-ui — 34 files, +991 -230

What changed

Structured JSON logging (default on):

  • LOG_FORMAT=json is the default. LOG_FORMAT=simple to opt out. Per-module levels: LOG_LEVEL_INDEXERS, LOG_LEVEL_ENRICHERS, LOG_LEVEL_RENDERERS. DEBUG_LOGGING=true still shortcut.
  • New: log_buffer.py (ring buffer + FileLogSink for persistent JSONL), log_formatter.py (JSON formatter)
  • Uvicorn, config reloader, and CLI client all use JSON now

Log API + Web UI:

  • GET /explorer/api/logs?level=&phase=&limit= — query ring buffer
  • GET /explorer/api/logs/download — full JSONL history
  • /health/ now includes phase_durations + recent_errors
  • New "Logs" tab in explorer (2nd tab): filters, auto-refresh, compact terminal-style rows, template error highlighting
  • "Recent Issues" card on home page. Warnings tile counts from ring buffer (matches Logs tab). Clickable tile links to pre-filtered Logs tab.

Log level hygiene:

  • 51 logger.warning to logger.info (fallbacks, skipping, graceful conditions)
  • 15 logger.warning to logger.debug (SLX name shortening, qualifier resolution, diagnostic detail)
  • Remaining WARNING calls are genuine (auth failures, git errors, connection issues)

Fixes:

  • "All Kinds" sidebar now shows kind names (was using wrong key: k.kind now k.artifact_kind)
  • Test grep pattern updated for JSON log format
  • Warnings count now from ring buffer, matching Logs tab display
  • Upload timing: "Workspace builder data uploaded successfully in 2.3s"

Docs: stuck.md, cloudquery-debug-logging.md, workspace-builder.md, README.md

@stewartshea
stewartshea merged commit ebca5a0 into main Jul 31, 2026
20 of 21 checks passed
@stewartshea
stewartshea deleted the logging-refactor-structured-logs-ui branch July 31, 2026 18:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant