Add log buffer and reduce info log noise - #814
Merged
Merged
Conversation
stewartshea
commented
Jul 31, 2026
Contributor
- Introduce a thread-safe ring buffer to store recent log entries
- Capture rendering warnings/errors in the buffer for UI display
- Add a "Logs" tab to the explorer and a "Recent Issues" card to the home page
- Add phase durations and recent errors to the health endpoint
- Switch frequent discovery and resolution logs from info to debug level
- Add per-module log level configuration via environment variables
- Introduce a thread-safe ring buffer to store recent log entries - Capture rendering warnings/errors in the buffer for UI display - Add a "Logs" tab to the explorer and a "Recent Issues" card to the home page - Add phase durations and recent errors to the health endpoint - Switch frequent discovery and resolution logs from info to debug level - Add per-module log level configuration via environment variables
Introduce JSON console logging as the default format (`LOG_FORMAT=json`), per-module level control via `LOG_LEVEL_INDEXERS`, `LOG_LEVEL_ENRICHERS`, `LOG_LEVEL_RENDERERS`, and `LOG_LEVEL_WORKSPACE_BUILDER`, a new Logs tab in the Workspace Explorer with live filtering, a `/explorer/api/logs` endpoint for programmatic access, and an enhanced `/health/` endpoint reporting phase durations and recent errors. Update documentation throughout to describe the new logging features, the UI Logs tab, and the workbench builder's page updates.
The CLI now uses StructuredJsonFormatter for machine-readable output, consistent with the server. The explorer page renders log entries with a structured layout and corrects field references from `kind` to `artifact_kind` to match the updated data model.
These are expected conditions (e.g., missing configuration, lookup failures) that don't warrant warnings. Also fixes a regex in a GCP test to properly match the backend selection message.
Also fix a test assertion in the GCP/K8s test to use the correct log file and simplify the regex pattern. Update the home dashboard to fetch the warning count from the API instead of relying on the last run's stored value.
stewartshea
commented
Jul 31, 2026
stewartshea
left a comment
Contributor
Author
There was a problem hiding this comment.
Structured Logging Refactor — Review
Branch: logging-refactor-structured-logs-ui
34 files, +991 −230
What changed
New modules (2 files):
src/workspace_builder/log_buffer.py— Thread-safe ring buffer (500 entries) +FileLogSinkthat persists full log history to${RW_LOG_FILE:-/tmp/runwhen-logs.jsonl}src/workspace_builder/log_formatter.py—StructuredJsonFormatter:{"timestamp","level","message","logger","module"}
Structured JSON logging (default on):
src/workspace_builder/startup.py—LOG_FORMAT=jsonis the default console format. Per-module log levels viaLOG_LEVEL_INDEXERS,LOG_LEVEL_ENRICHERS,LOG_LEVEL_RENDERERS,LOG_LEVEL_WORKSPACE_BUILDER.DEBUG_LOGGING=truestill works as shortcut. Uvicorn loggers also configured for JSON. Ring buffer + file sink always active.src/run.py— CLI client now wiresStructuredJsonFormatterso its output matches the server processsrc/config_reloader.py— Kubeconfig reloader uses JSON formatsrc/azure_utils.py,src/k8s_utils.py—print()→logger.info/warning/error
Log API + Web UI:
GET /explorer/api/logs?level=&phase=&limit=— Query the ring bufferGET /explorer/api/logs/download— Download full JSONL history/health/— Now includesphase_durations+recent_errorsexplorer.html— New "Logs" tab (2nd tab) with level/phase filters, auto-refresh, terminal-style compact rows, template error highlightinghome.html— "Recent Issues" card with real ERROR content, warnings tile counts from ring buffer (matches Logs tab), clickable → pre-filtered Logs tab
Log level hygiene (~90 demotions):
- 51
logger.warning→logger.info(expected fallbacks/skipping/graceful conditions) - 15
logger.warning→logger.debug(diagnostic detail, SLX name shortening, qualifier resolution) - 1
logger.info→logger.debug(per-item rendering) - Remaining WARNING calls are genuine problems (auth failures, git errors, connection issues)
"All Kinds" fix:
explorer.htmlsidebar used wrong keyk.kind→ fixed tok.artifact_kind(5 locations)
Tests:
.test/gcp/gcp-and-k8s/Taskfile.yaml— Grep pattern updated for JSON log format
Docs (4 files):
docs/user-guide/troubleshooting/stuck.md— Web UI Logs Tab, Logs API, Structured JSON Logging sectionsdocs/user-guide/troubleshooting/cloudquery-debug-logging.md— Expanded to full logging configuration referencedocs/user-guide/features/workspace-builder.md— Logs tab + Recent Issues documentationREADME.md— Logs tab + structured logging bullets
How to use
# Structured JSON is the DEFAULT — no flags needed
./run.sh
# Per-module debug (only indexers, everything else stays at INFO)
LOG_LEVEL_INDEXERS=DEBUG ./run.sh
# Plain-text format (opt-out)
LOG_FORMAT=simple ./run.sh
# Download full log history
curl http://localhost:8000/explorer/api/logs/download > runwhen-logs.jsonl
# Query recent errors
curl "http://localhost:8000/explorer/api/logs?level=ERROR&limit=10"Verify
# Logs should be valid JSON
curl -s http://localhost:8000/explorer/api/logs?limit=1 | jq .entries[0].timestamp
# Health should include timing
curl -s http://localhost:8000/health/ | jq .phase_durations
Contributor
Author
Structured Logging RefactorBranch: What changedStructured JSON logging (default on):
Log API + Web UI:
Log level hygiene:
Fixes:
Docs: stuck.md, cloudquery-debug-logging.md, workspace-builder.md, README.md |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.