Add optional TTL for migrated Cassandra timeseries data - #12
Open
hophead12 wants to merge 2 commits into
Open
Conversation
Migrated data currently lives forever in Cassandra since no TTL is set on writes. Add an optional -ttl CLI argument (in days) that applies USING TTL to ts_kv_cf and ts_kv_partitions_cf inserts. ts_kv_latest_cf is intentionally left unaffected, matching ThingsBoard's own TTL semantics where only historical points expire, not the latest value. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Reject non-positive ttl values and ttl values exceeding Cassandra's hard-coded 20 year (7300 day) TTL maximum, instead of letting them fail later with an opaque error from CQLSSTableWriter or silently producing no TTL (ttl=0 means "no TTL" in Cassandra). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Migrated timeseries data currently lives in Cassandra forever — none of the generated SSTables set a TTL, and the target tables have no
default_time_to_live. For teams that only need historical data for a limited retention window, this means migrated data has to be cleaned up manually afterwards, or it just accumulates indefinitely and wastes disk space.This PR adds an optional
-ttl <days>CLI argument so migrated data can expire automatically in Cassandra, the same way it would if it had been written with a TTL policy from the start.Why this matters
-ttl 90and Cassandra will expire the migrated rows automatically via native TTL, no follow-up cleanup step required.What changed
-ttl(integer, days). If omitted, behavior is unchanged — no TTL, data never expires.USING TTL <seconds>tots_kv_cf(historical points) andts_kv_partitions_cf(partition bookkeeping).ts_kv_latest_cf(last known value per key) is intentionally not affected by-ttl, matching ThingsBoard's own TTL semantics where only historical points expire, not the current value.-ttlmust be a positive number of days and cannot exceed 7300 days (20 years) — Cassandra's hard-coded maximum TTL. Values outside this range are now rejected with a clear error instead of failing later with an opaque error fromCQLSSTableWriter, or silently doing nothing (ttl=0means "no TTL" in Cassandra).Testing
Ran an end-to-end migration against a real ThingsBoard Postgres database (Docker, ~8.4M
ts_kvrows) using the documented dump commands, then ran the migrator with-ttl 30:org.apache.cassandra.tools.SSTableExport/SSTableMetadataViewer: every row ints_kv_cfandts_kv_partitions_cfhasttl = 2592000(30 days) with a correctexpires_at.ts_kv— the small difference is expected (rows referencing entities missing fromrelated_entities.dmpare skipped by design).mvn clean compile assembly:singlebuilds cleanly.Note (separate, pre-existing issue found during testing)
The packaged jar has no SLF4J binding (only
slf4j-api/log4j-over-slf4j), so all of the tool's logging (Lines processed,Lines migrated, errors) is silently dropped at runtime (SLF4J: Defaulting to no-operation (NOP) logger implementation). This is unrelated to this change and not fixed here, but worth a follow-up since it breaks the documented-linesToSkipresume workflow, which relies on reading progress from the log.