Skip to content

Add optional TTL for migrated Cassandra timeseries data - #12

Open
hophead12 wants to merge 2 commits into
thingsboard:masterfrom
hophead12:feature/ts-ttl-option
Open

Add optional TTL for migrated Cassandra timeseries data#12
hophead12 wants to merge 2 commits into
thingsboard:masterfrom
hophead12:feature/ts-ttl-option

Conversation

@hophead12

Copy link
Copy Markdown

Summary

Migrated timeseries data currently lives in Cassandra forever — none of the generated SSTables set a TTL, and the target tables have no default_time_to_live. For teams that only need historical data for a limited retention window, this means migrated data has to be cleaned up manually afterwards, or it just accumulates indefinitely and wastes disk space.

This PR adds an optional -ttl <days> CLI argument so migrated data can expire automatically in Cassandra, the same way it would if it had been written with a TTL policy from the start.

Why this matters

  • Without this feature: every row written by this tool is permanent. If a customer only needs, say, 90 days of historical data available after migration, there is no way to enforce that at write time — it either has to be deleted manually with a separate maintenance job, or it silently stays forever and grows the cluster's storage footprint.
  • With this feature: operators can pass -ttl 90 and Cassandra will expire the migrated rows automatically via native TTL, no follow-up cleanup step required.

What changed

  • New optional CLI option -ttl (integer, days). If omitted, behavior is unchanged — no TTL, data never expires.
  • TTL is applied via USING TTL <seconds> to ts_kv_cf (historical points) and ts_kv_partitions_cf (partition bookkeeping).
  • ts_kv_latest_cf (last known value per key) is intentionally not affected by -ttl, matching ThingsBoard's own TTL semantics where only historical points expire, not the current value.
  • Validation: -ttl must be a positive number of days and cannot exceed 7300 days (20 years) — Cassandra's hard-coded maximum TTL. Values outside this range are now rejected with a clear error instead of failing later with an opaque error from CQLSSTableWriter, or silently doing nothing (ttl=0 means "no TTL" in Cassandra).
  • README updated with a new "Setting TTL for migrated data" section.

Testing

Ran an end-to-end migration against a real ThingsBoard Postgres database (Docker, ~8.4M ts_kv rows) using the documented dump commands, then ran the migrator with -ttl 30:

java -Xmx5g -jar database-migrator-1.0-SNAPSHOT-jar-with-dependencies.jar \
    -telemetryFrom ts_kv_all.dmp -relatedEntities related_entities.dmp -dictionary ts_kv_dictionary.dmp \
    -tsOut ts_kv_cf -partitionsOut ts_kv_partitions_cf -castEnable false -partitioning MONTHS -ttl 30
  • Process completed successfully, exit code 0.
  • Verified TTL was actually written into the generated SSTables using the bundled org.apache.cassandra.tools.SSTableExport / SSTableMetadataViewer: every row in ts_kv_cf and ts_kv_partitions_cf has ttl = 2592000 (30 days) with a correct expires_at.
  • Row count sanity check: 8,430,969 rows migrated vs 8,431,293 rows in source Postgres ts_kv — the small difference is expected (rows referencing entities missing from related_entities.dmp are skipped by design).
  • mvn clean compile assembly:single builds cleanly.

Note (separate, pre-existing issue found during testing)

The packaged jar has no SLF4J binding (only slf4j-api/log4j-over-slf4j), so all of the tool's logging (Lines processed, Lines migrated, errors) is silently dropped at runtime (SLF4J: Defaulting to no-operation (NOP) logger implementation). This is unrelated to this change and not fixed here, but worth a follow-up since it breaks the documented -linesToSkip resume workflow, which relies on reading progress from the log.

Danylo Bosenko and others added 2 commits August 18, 2026 17:03
Migrated data currently lives forever in Cassandra since no TTL is set
on writes. Add an optional -ttl CLI argument (in days) that applies
USING TTL to ts_kv_cf and ts_kv_partitions_cf inserts. ts_kv_latest_cf
is intentionally left unaffected, matching ThingsBoard's own TTL
semantics where only historical points expire, not the latest value.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Reject non-positive ttl values and ttl values exceeding Cassandra's
hard-coded 20 year (7300 day) TTL maximum, instead of letting them
fail later with an opaque error from CQLSSTableWriter or silently
producing no TTL (ttl=0 means "no TTL" in Cassandra).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant