Audit/reproducibility types: emit seed and RNG state per record

[mferretti]

Summary

Two new field types for audit / reproducibility, so a generated record can carry the information needed to trace and re-generate it:

1. seed — emit the seed that produced the record.
2. rngstate (working name) — emit the RNG state at the point the record was produced, so the exact value(s) can be re-generated later.

Opening this to capture the design questions and the grounded technical findings, then discuss here before implementing.

---

Current seeding architecture (grounded in code)

• core/.../seed/RandomProvider.java
  • one global masterSeed (from job config / CLI --seed).
  • per worker thread: deriveSeed(masterSeed, workerId) where workerId is a logical id (0,1,2…) assigned via AtomicInteger on first getRandom() — deliberately not the JVM thread id (see class Javadoc), so runs are reproducible.
  • each worker gets new Random(threadSeed) — plain java.util.Random (48-bit LCG, state is a single long), held in a ThreadLocal.
• generators/.../semantic/FakerCache.java
  • new Faker(locale, THREAD_RANDOM.get()) — Datafaker holds a reference to our thread-local Random. So primitive generators and Datafaker draw from the same per-worker stream.

Consequences for this feature:

• The reproducibility coordinates of a record are essentially (masterSeed, workerId, draw-position-in-that-worker's-stream).
• threadSeed is fully derivable from (masterSeed, workerId) via deriveSeed, so it is redundant with those two.

---

Type 1 — seed

Question: global seed, thread seed, or two types?

Observations:

• The global masterSeed is constant for the whole run — already known from the job config/CLI. Emitting it per record is cheap but low-information (same on every row).
• The per-worker derived seed (or equivalently the workerId) is the bit that varies and actually identifies which deterministic sub-stream produced the row.

Proposal to discuss:

• A single seed type with a selector, e.g. seed[master] and seed[worker] (and maybe seed[worker_id]), rather than two separate types — keeps the type system smaller.
• Minimum useful audit datum = masterSeed + workerId; from those the worker's Random start state is reconstructable. threadSeed could be a convenience output but is derivable.

Open: do we want all three (master / worker_id / derived_thread_seed), or just master + worker_id?

---

Type 2 — rngstate (emit RNG state to re-generate the value)

Main question: can we, in Java, restore a Random to a state that reproduces the same value?

Yes, for java.util.Random (which is what this project uses):

• Its entire state is a single 48-bit value in the private seed (AtomicLong) field.
• Capture: read that field (reflection) — or better, use a small Random subclass exposing getState()/setState().
• Restore caveat: setSeed(s) does not restore raw state — the constructor/setSeed apply initialScramble ((s ^ 0x5DEECE66DL) & mask). To restore exact state you must write the raw 48-bit internal value back (reflection or the subclass), not call setSeed.
• After restoring, the next nextX() reproduces the same draw. Confirmed feasible.
• (SplittableRandom would be harder — no state accessors. Sticking with java.util.Random keeps this tractable. A purpose-built Random subclass is the cleanest path and avoids reflection.)

Can we do it for Datafaker?

• Likely yes, because Faker holds a reference to our Random (see FakerCache). If we restore that Random's internal state to the point immediately before a Faker call and replay the same call, Datafaker should consume the same draws and produce the same fake value — Datafaker value generation is a function of the injected Random.
• Caveats to validate (the discussion meat):
  1. Capture granularity. A record has many fields; each field consumes a variable number of draws (random-length strings, regex, etc.). One state snapshot before the whole record reproduces the entire record only if all fields are replayed in the same order. Reproducing a single field needs a per-field snapshot. Which granularity do we want to emit — per record, or per field?
  2. Shared interleaved stream. Primitives and Faker share the worker's Random. Reproduction must replay the same interleaving of primitive + Faker draws.
  3. Non-RNG state in Datafaker providers. Features like unique/distinct, options caches, or any provider holding internal mutable state beyond the Random would not be reproduced by RNG-state restore alone. Need to confirm which providers (if any) we use that violate "pure function of the injected Random."
  4. Storage form. The state is a 48-bit long → emit as a long/hex string. Cheap.
  5. Thread-safety of capture. Snapshotting mid-generation on the worker thread (synchronous to generation) is fine; doing it off-thread is not.

---

Open questions for discussion

1. seed type: one type with master/worker/worker_id selectors, or separate types? Which outputs are actually worth emitting given threadSeed is derivable?
2. rngstate: capture granularity — per record (reproduce the whole row) vs per field (reproduce one value)? This drives where we snapshot.
3. Implementation of state access: dedicated Random subclass with getState()/setState() (preferred, no reflection) vs reflection on java.util.Random.seed.
4. Verify Datafaker providers in use are pure functions of the injected Random (no unique/options/internal caches) — otherwise restore won't reproduce.
5. Interaction with the container/CI work: this is exactly the "record self-documents what produced it" idea parked in docs/internal/CONTAINER-CICD-IDEAS.md (Future enhancements) — supersedes the earlier "echo the resolved seed to logs" half-measure.

---

Refs: core/src/main/java/com/datagenerator/core/seed/RandomProvider.java, generators/src/main/java/com/datagenerator/generators/semantic/FakerCache.java.