Skip to content

Feature/scitt scrapi lifecycle#127

Closed
howethomas wants to merge 14 commits intomainfrom
feature/scitt-scrapi-lifecycle
Closed

Feature/scitt scrapi lifecycle#127
howethomas wants to merge 14 commits intomainfrom
feature/scitt-scrapi-lifecycle

Conversation

@howethomas
Copy link
Contributor

@howethomas howethomas commented Mar 2, 2026

Note

Medium Risk
Moderate risk: adds new public monitoring endpoint and new outbound integrations (SCRAPI registration, webhooks, vfun) that can affect reliability and data exposure if misconfigured, but core storage paths are only lightly refactored (indexing optimization).

Overview
Adds end-to-end operational tooling and new pipeline links: a SCRAPI-based links.scitt lifecycle registrar that signs the vCon hash, posts it to a transparency service, and optionally stores a scitt_receipt analysis entry (with accompanying unit tests and updated README docs).

Introduces a new links.wtf_transcribe integration for vfun that supports multi-instance failover with health tracking, Redis-backed transcription caching, and WTF-format analysis output; adds a keyword_tagger link for tagging vCons based on transcription keywords.

Improves observability/deployment by adding a SigNoz docker-compose stack plus OpenTelemetry env examples, switching apt sources to HTTPS in the Dockerfile, and adding scripts/docs for NAS-based transcription stress/performance testing.

API changes include a new unauthenticated /stats/queue endpoint for Redis list depth and an ingest-path indexing optimization that indexes parties directly (index_vcon_parties) to avoid redundant Redis reads; webhook payloads are normalized to vCon version 0.3.0 (both link and new storage.webhook).

Written by Cursor Bugbot for commit d6f2daa. Configure here.

howethomas and others added 14 commits January 27, 2026 22:24
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Updates webhook link to set vcon version to 0.3.0 for
compatibility with vcon-mcp REST API.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Configure apt to use HTTPS sources for environments
where HTTP port 80 is blocked.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Includes docker-compose and config files for SigNoz
observability stack with OpenTelemetry collector.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Public endpoint (no auth) that returns the depth of any Redis list,
used by the audio adapter for backpressure control.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The post_vcon and external_ingress_vcon paths called index_vcon() which
re-read the vCon from Redis (JSON.GET) and duplicated the sorted set add
(ZADD) that was already done by the caller. This added 2 unnecessary
Redis round-trips per ingest.

Extract index_vcon_parties() that takes the vCon dict directly, and use
it in both POST paths. The original index_vcon() is preserved for the
bulk re-indexing endpoint. Reduces ingest from 11 to 9 Redis ops per
vCon, measured 4.9x improvement in adapter posting throughput.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The supabase_webhook was running as a sequential chain link, blocking
each worker for ~560ms per vCon. By moving it to a storage slot, the
webhook now executes post-chain in parallel via ThreadPoolExecutor,
reducing per-vCon P50 latency from 617ms to 123ms (5x improvement).

New module server/storage/webhook/ wraps the existing HTTP POST logic
with the storage save() interface.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The wtf_transcribe link had no retry logic — a single vfun failure
silently dropped the transcription. This adds:

- _VfunHealthTracker: thread-safe singleton tracking instance health
  across all workers, with 30-second self-healing recovery window
- get_vfun_urls(): returns URLs in priority order (healthy shuffled,
  then recovering oldest-first, then down instances)
- Fallback loop: tries all configured vfun instances before giving up
- Redis transcription cache: skips vfun calls for previously transcribed
  audio files (7-day TTL)

On failure, instances are marked DOWN and bypassed until the recovery
window expires, then automatically retried and restored on success.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- SignOz OTEL collector config and docker-compose integration
- Performance testing and vfun crash/stress test reports
- Utility scripts for NAS pipeline operations and debugging

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace the DataTrails-specific OIDC authentication and registration
with generic SCRAPI calls to SCITTLEs (self-hosted transparency service).
Register vcon_created and vcon_enhanced lifecycle events per
draft-howe-vcon-lifecycle, storing COSE receipts as scitt_receipt
analysis entries on each vCon.

- Remove OIDC_Auth class and DataTrails-specific endpoints
- Add register_statement() with sync (201) and async (303) SCRAPI handling
- Add wait_for_entry_id() polling and get_receipt() for async flow
- Store receipt metadata (entry_id, vcon_operation, vcon_hash) on vCon
- Add 12 unit tests covering registration, polling, and link runner
- Document SCITT Lifecycle Registration in README

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The conserver's __init__.py uses ``from links.scitt import ...`` which
registers submodules under ``links.scitt.*`` in sys.modules, while
pytest imports create a parallel ``server.links.scitt.*`` entry.
Patching the wrong module object meant time_sleep, requests.get, and
VconRedis mocks had no effect — tests hit real services and took 80s.

Fixes:
- Use ``links.scitt.register_signed_statement`` path for submodule
  attribute mocks (time_sleep, requests.get/post)
- Use ``links.scitt.create_hashed_signed_statement`` for COSE mocks
- Use ``server.links.scitt`` for __init__.py namespace names (VconRedis)
- Replace http://scittles:8000 with non-routable RFC 6761 URL
  (http://scrapi.test.invalid:9999) as safety net
- Test suite now runs in 0.5s instead of 80s

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 3 potential issues.

Bugbot Free Tier Details

Your team is on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle for each member of your team.

To receive Bugbot reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial.

Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Comment @cursor review or bugbot run to trigger another review on this PR


return res["operationID"]
else:
response.raise_for_status()
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Function returns None for unexpected success status codes

High Severity

register_statement implicitly returns None when the SCRAPI service responds with a 2xx status code other than 201 (e.g. 200). raise_for_status() only raises for 4xx/5xx codes, so a 200 response falls through without raising or returning. The caller in __init__.py immediately accesses result['entry_id'], which would crash with a TypeError on NoneType.

Additional Locations (1)

Fix in Cursor Fix in Web

return JSONResponse(content={"list_name": list_name, "depth": depth})
except Exception as e:
logger.error(f"Error getting queue depth for '{list_name}': {str(e)}")
raise HTTPException(status_code=500, detail="Failed to get queue depth")
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unauthenticated endpoint exposes arbitrary Redis list lengths

Medium Severity

The /stats/queue endpoint is registered on app directly (not api_router), so it bypasses the Security(get_api_key) dependency that protects all other data endpoints. It accepts an arbitrary list_name parameter and queries Redis with llen, allowing unauthenticated users to probe the existence and length of any Redis list by name.

Fix in Cursor Fix in Web


# Other Content
"other": {
"profanity": ["fuck", "shit", "damn", "ass"],
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Substring matching causes widespread false positive keyword tags

Medium Severity

The find_keywords function uses Python's in operator for substring matching, and several keywords are short common substrings. Most notably, "ass" in the profanity list will match "class", "pass", "assist", "classic", "passage", "ambassador", etc. Similarly "damn" matches "damage". In phone call transcriptions, these words are extremely common, causing nearly every vCon to be falsely tagged as profanity.

Additional Locations (1)

Fix in Cursor Fix in Web

@howethomas howethomas closed this Mar 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant