Skip to content

Optimize vCon ingest: index by parties only, remove default TTL#134

Merged
pavanputhra merged 3 commits intomainfrom
feature/optimize-ingest-indexing
Mar 8, 2026
Merged

Optimize vCon ingest: index by parties only, remove default TTL#134
pavanputhra merged 3 commits intomainfrom
feature/optimize-ingest-indexing

Conversation

@howethomas
Copy link
Contributor

Summary

Optimizes vCon ingest by removing redundant Redis operations and removing the default TTL on newly stored vCons. Cherry-picked and consolidated from the dev server consolidation branches (see BRANCH_CONSOLIDATION_PLAN.md).

Changes

1. Optimize vCon ingest (fewer Redis round-trips)

Before: On each ingest (post_vcon and external_ingress_vcon), the code called index_vcon(uuid), which re-read the vCon from Redis (JSON.GET) and called add_vcon_to_set again even though the caller had already done both. That added 2 unnecessary Redis operations per vCon.

After: Ingest paths no longer call index_vcon(). They use the in-memory vCon and only run the party-indexing logic via a new helper. Sorted-set add is done once in the caller. Result: 2 fewer Redis ops per ingest; measured ~4.9x improvement in adapter posting throughput.

2. Index by parties only – index_vcon_parties()

  • New helper: index_vcon_parties(vcon_uuid, parties) – indexes only by party (tel, mailto, name). No Redis read, no sorted-set add.
  • Ingest paths call index_vcon_parties(str(inbound_vcon.uuid), dict_vcon["parties"]) so indexing no longer does an extra JSON.GET.
  • Bulk re-indexing still uses index_vcon(uuid), which now reads from Redis, adds to the sorted set, then calls index_vcon_parties(). Behavior unchanged for re-indexing.

Search behavior (index by party tel/mailto/name with VCON_INDEX_EXPIRY) is unchanged; only the ingest path avoids the extra read and duplicate ZADD.

3. Remove default vCon TTL

Before: After storing a vCon, the code called expire(key, VCON_REDIS_EXPIRY) (e.g. 3600s). Every new vCon had a 1-hour TTL and would disappear from Redis unless something else extended or removed it.

After: That expire() call is removed in both post_vcon and external_ingress_vcon. Newly stored vCons no longer get a default TTL; they remain in Redis until something explicitly sets TTL or deletes them. Docstrings updated accordingly.

Rationale: Avoids vCons expiring from Redis when retention should be controlled by storage backends or other logic instead of an implicit 1-hour default.

Effect

Change Effect
Optimize ingest 2 fewer Redis ops per ingest; large throughput gain (~4.9x in testing).
Index by parties only Same search indexes; no extra JSON.GET on ingest.
Remove default TTL Stored vCons are not auto-expired; retention is explicit.

howethomas and others added 3 commits March 4, 2026 17:58
The post_vcon and external_ingress_vcon paths called index_vcon() which
re-read the vCon from Redis (JSON.GET) and duplicated the sorted set add
(ZADD) that was already done by the caller. This added 2 unnecessary
Redis round-trips per ingest.

Extract index_vcon_parties() that takes the vCon dict directly, and use
it in both POST paths. The original index_vcon() is preserved for the
bulk re-indexing endpoint. Reduces ingest from 11 to 9 Redis ops per
vCon, measured 4.9x improvement in adapter posting throughput.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Stop setting default expiry (VCON_REDIS_EXPIRY) on newly stored vCons in
post_vcon and external_ingress_vcon. Stored vCons no longer auto-expire
from Redis; retention is controlled by storage backends or explicit TTL.

Made-with: Cursor
- Add mock_redis.sadd = AsyncMock() so index_vcon_parties can run in tests
- Patch index_vcon_parties in expiry tests; update assertions for no default
  VCON_REDIS_EXPIRY on vcon key (only index keys get expire)
- Rename expiry tests to reflect new behavior (stores without default expiry)
- Replace test_external_ingress_expire_called_before_rpush with
  test_external_ingress_adds_to_ingress_list

Made-with: Cursor
@pavanputhra pavanputhra merged commit a2fc92d into main Mar 8, 2026
1 check passed
@pavanputhra pavanputhra deleted the feature/optimize-ingest-indexing branch March 8, 2026 23:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants