FE-1130: Token frame format v2 — packed struct token layout

[kube]

🌟 What is the purpose of this PR?

Replaces the simulation engine's uniform one-Float64-slot-per-dimension token encoding with a schema-driven packed struct per token (format v2), everywhere: interactive engine, Monte Carlo, frame readers, and the encoding playground. Booleans now occupy a single byte instead of eight, places are byte-addressed, and the per-colour layout module gives future wider types (128-bit UUIDs, FE-1121) a place to slot in as u64×2 fields.

🔗 Related links

• FE-1130: Petrinaut: token frame format v2 — packed struct token layout (internal)
• FE-1121: Add UUID discrete type to Petrinaut (internal — the design notes there build on this layout machinery)
• Stacked on FE-1129: Token encoding playground (Storybook) #8943 (playground), which stacks on H-6519: Add discrete token attribute types #8764 (discrete types).

🚫 Blocked by

• H-6519: Add discrete token attribute types #8764
• FE-1129: Token encoding playground (Storybook) #8943 (base branch — merges first)

🔍 What does this change?

• New engine/token-layout.ts — single source of truth for the physical layout: real/integer → f64 (8 B), boolean → u8 (1 B); fields ordered by decreasing alignment (stable), stride rounded up to 8 B so every f64 field stays addressable through a shared Float64Array view (no DataView in hot paths). Read/write goes through readTokenRecord / writeTokenValue / encodeTokenToBytes only.
• EngineFrame bumped to FRAME_VERSION = 2: per-place state is byte-addressed ({ byteOffset, count, strideBytes }), the header records the token region's byte length, snapshots carry Uint8Arrays, and all whole-token moves (kernel-produced insertions, removal/compaction) are byte-range copies.
• Dynamics integrate real fields only: Euler walks realFieldF64Offsets; discrete bytes are copied through untouched (structurally stronger than zeroing derivatives).
• Determinism preserved: kernels sample distributions in element declaration order with the seeded RNG, then pack per layout — same seed, same run.
• Monte Carlo frame buffers use byte capacities (tokenByteCount / tokenByteCapacity, same ×2 growth policy, dual u8/f64 views).
• Public API break: getPlaceTokenValues / SimulationPlaceTokenValues removed — raw f64 access is meaningless under mixed field widths; consumers use getPlaceTokens (typed records). The monorepo was swept: no usage outside the petrinaut packages.
• Playground drops its v1 mode and reuses the core layout module, so the story renders the real engine layout.
• simulation/ARCHITECTURE.md and engine/README.md token-storage sections rewritten for the packed layout.

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

• modifies an npm-publishable library and I have added a changeset file(s)
  • (token-frame-format-packed-structs.md — dedicated entry for the layout change and the API slimming)

📜 Does this require a change to the docs?

The changes in this PR:

• are internal and do not require a docs change
  • (frame format internals; user-facing behaviour is unchanged — user docs were updated for typed dimensions in H-6519: Add discrete token attribute types #8764)

🕸️ Does this require a change to the Turbo Graph?

The changes in this PR:

• do not affect the execution graph

⚠️ Known issues

• Engine throughput has not been benchmarked v1 vs v2. Nothing suggests a regression (element-wise copy loops became bulk byte copies), but there is no measurement.
• Frames are ephemeral per run, so no data migration is needed; the version bump makes any stale-frame decode fail loudly rather than silently misread.

🐾 Next steps

• FE-1121: UUID as a u64×2 field (align 8) with bigint at the JS boundary — the layout machinery here is built for it.
• Possible follow-up: authoring-time warning when a place has dynamics enabled but its colour has no real elements (currently a silent no-op by design).

🛡 What tests cover this?

• New engine/token-layout.test.ts (layout computation, encode/decode round-trips incl. u8 booleans and integer rounding) + a discrete-fields-untouched dynamics integration test
• All engine/Monte Carlo/frame-reader suites updated to the byte-addressed fixtures — 553 tests in @hashintel/petrinaut-core, 137 in @hashintel/petrinaut
• lint:tsc (tsgo) and lint:eslint (oxlint) clean for both packages

❓ How to test this?

1. Checkout the branch, yarn dev in libs/@hashintel/petrinaut.
2. Run a Quick Simulation on a net with typed tokens (include a Boolean dimension); confirm initial state, transitions, dynamics, visualizers, and the timeline behave as before.
3. Run an Experiment (Monte Carlo) with metrics; confirm progress and metric charts.
4. Open Dev / Token Encoding Playground: a Boolean dimension renders as one u8 byte with hatched tail padding, sizeof(token) reflects the packed stride.

📹 Demo

See the token-encoding playground story: the memory view renders the exact v2 layout this PR ships (u8 booleans, alignment padding, byte-addressed slots).

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
hash	Ready	Preview, Comment	Jul 7, 2026 12:48pm
petrinaut	Ready	Preview, Comment	Jul 7, 2026 12:48pm

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
hashdotdesign-tokens	Ignored	Preview	Jul 7, 2026 12:48pm

[cursor]

PR Summary

High Risk
This is a breaking internal frame format and public reader API change across the simulation engine, transitions, dynamics, and Monte Carlo—incorrect layout or alignment would corrupt runs, though version checks and broad test updates mitigate that.

Overview
Introduces engine/token-layout.ts as the layout source of truth and moves the internal EngineFrame to FRAME_VERSION = 2: token data lives in a Uint8Array region of per-colour packed structs (real/integer → f64, boolean → u8, stride rounded to 8 bytes), with places tracked as byteOffset / strideBytes instead of flat f64 slots.

The stepping path (initial marking, dynamics, transition read/write, add/remove compaction) and Monte Carlo buffers are rewired to copy and integrate byte blocks; Euler dynamics only update real fields via realFieldF64Offsets, leaving integers and booleans unchanged.

Public API: removes getPlaceTokenValues / SimulationPlaceTokenValues; getPlaceTokens(place) no longer takes a colour argument (type comes from the place), and buildMetricState drops the types list. Compiled transitions carry tokenLayout instead of elementNames. Docs and tests are updated for the byte layout; a changeset marks patch releases for core and petrinaut.

^{Reviewed by Cursor Bugbot for commit e784cb6. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Copilot]

Pull request overview

This PR upgrades Petrinaut’s simulation frame token storage to frame format v2: a packed-struct, byte-addressed token region with a schema-driven per-colour layout (f64 for real/integer, u8 for boolean), and updates the engine, Monte Carlo pipeline, frame readers, and the token encoding playground to use that layout consistently.

Changes:

• Introduces a core TokenSlotLayout abstraction and helpers for computing layouts and reading/writing packed token bytes (computeTokenSlotLayout, readTokenRecord, encodeTokenToBytes, etc.).
• Bumps engine frames to version 2 and converts all token-region handling from Float64Array value slots to Uint8Array byte regions with shared u8/f64 views.
• Updates Monte Carlo buffering/transition effects/dynamics and the dev token encoding playground to reflect the packed layout; removes the now-meaningless raw f64 token-values API.

Reviewed changes

Copilot reviewed 44 out of 45 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
libs/@hashintel/petrinaut/src/ui/dev/token-encoding-playground/token-memory-view.tsx	Adapts the memory view UI to render TokenSlotLayout fields (byte offsets/sizes and field metadata) from core.
libs/@hashintel/petrinaut/src/ui/dev/token-encoding-playground/token-encoding-playground.tsx	Removes v1/v2 toggle and scopes Monaco’s built-in TS language service to the playground lifecycle.
libs/@hashintel/petrinaut/src/ui/dev/token-encoding-playground/playground-monaco.ts	Adds enable/disable controls for Monaco TS mode configuration and clears TS markers on unmount.
libs/@hashintel/petrinaut/src/ui/dev/token-encoding-playground/physical-layout.ts	Replaces local layout/codec implementation with thin wrappers over core token-layout encode/decode helpers.
libs/@hashintel/petrinaut/src/ui/dev/token-encoding-playground/physical-layout.test.ts	Updates playground layout and encode/decode tests to use core TokenSlotLayout.
libs/@hashintel/petrinaut/src/ui/dev/token-encoding-playground/monaco-typescript.d.ts	Extends the local TS defaults typing to include setModeConfiguration.
libs/@hashintel/petrinaut/src/react/playback/provider.test.tsx	Updates playback mocks to match the removal of getPlaceTokenValues.
libs/@hashintel/petrinaut/src/main.ts	Removes the exported SimulationPlaceTokenValues type from the Petrinaut package surface.
libs/@hashintel/petrinaut-core/src/simulation/runtime/simulation.test.ts	Updates frame construction in tests to use an empty Uint8Array token buffer.
libs/@hashintel/petrinaut-core/src/simulation/monte-carlo/types.ts	Renames Monte Carlo token capacity/count fields to byte-based equivalents.
libs/@hashintel/petrinaut-core/src/simulation/monte-carlo/transition-effect.ts	Switches transition effects to read/write packed token bytes via token-layout helpers.
libs/@hashintel/petrinaut-core/src/simulation/monte-carlo/run-state.ts	Converts frame resizing and run summaries to use token byte counts/capacities.
libs/@hashintel/petrinaut-core/src/simulation/monte-carlo/monte-carlo-simulator.test.ts	Updates Monte Carlo tests to assert byte-sized token regions and to use typed token access.
libs/@hashintel/petrinaut-core/src/simulation/monte-carlo/internal-types.ts	Changes transition additions to store Uint8Array token blocks rather than number arrays.
libs/@hashintel/petrinaut-core/src/simulation/monte-carlo/frame-reader.ts	Updates Monte Carlo frame reader to return typed token records from packed bytes.
libs/@hashintel/petrinaut-core/src/simulation/monte-carlo/frame-operations.ts	Converts dynamics/removal/addition operations to work on packed byte regions.
libs/@hashintel/petrinaut-core/src/simulation/monte-carlo/frame-buffer.ts	Updates Monte Carlo frame buffer layout to include u8/f64 views over a byte-addressed token region.
libs/@hashintel/petrinaut-core/src/simulation/monte-carlo/advance-run.ts	Updates pending token-additions map type to Uint8Array[].
libs/@hashintel/petrinaut-core/src/simulation/index.ts	Removes SimulationPlaceTokenValues export from the simulation entrypoint.
libs/@hashintel/petrinaut-core/src/simulation/frames/internal-frame.ts	Bumps FRAME_VERSION to 2 and migrates internal frame layout/state to byte offsets + packed token bytes.
libs/@hashintel/petrinaut-core/src/simulation/frames/frame-reader.ts	Updates the main frame reader to decode typed tokens from packed bytes via readTokenRecord.
libs/@hashintel/petrinaut-core/src/simulation/frames/frame-reader.test.ts	Rewrites frame-reader tests to validate copied token records rather than copied f64 buffers.
libs/@hashintel/petrinaut-core/src/simulation/engine/types.ts	Updates engine types for byte-addressed dynamics inputs and includes per-place tokenLayout.
libs/@hashintel/petrinaut-core/src/simulation/engine/token-layout.ts	Adds the core packed token layout module (layout computation + encode/decode + shared views).
libs/@hashintel/petrinaut-core/src/simulation/engine/token-layout.test.ts	Adds unit tests covering layout math, encode/decode round-trips, and alignment checks.
libs/@hashintel/petrinaut-core/src/simulation/engine/token-layout.test-helpers.ts	Adds helpers for constructing/decoding v2 frames and token blocks in tests.
libs/@hashintel/petrinaut-core/src/simulation/engine/remove-tokens-from-simulation-frame.ts	Migrates token removal/compaction to operate on packed byte ranges.
libs/@hashintel/petrinaut-core/src/simulation/engine/remove-tokens-from-simulation-frame.test.ts	Rewrites token-removal tests around packed bytes and typed token decoding helpers.
libs/@hashintel/petrinaut-core/src/simulation/engine/README.md	Updates engine documentation to describe frame format v2 token storage and access patterns.
libs/@hashintel/petrinaut-core/src/simulation/engine/execute-transitions.ts	Updates transition execution to append packed token byte blocks and adjust byte offsets.
libs/@hashintel/petrinaut-core/src/simulation/engine/execute-transitions.test.ts	Updates transition execution tests to validate packed-token behavior via decode helpers.
libs/@hashintel/petrinaut-core/src/simulation/engine/compute-possible-transition.ts	Updates possible-transition computation to decode/encode packed token byte blocks and preserve determinism.
libs/@hashintel/petrinaut-core/src/simulation/engine/compute-possible-transition.test.ts	Updates possible-transition tests to validate packed byte outputs by decoding token blocks.
libs/@hashintel/petrinaut-core/src/simulation/engine/compute-place-next-state.ts	Updates dynamics stepping to integrate real fields only while copying discrete bytes through unchanged.
libs/@hashintel/petrinaut-core/src/simulation/engine/compute-place-next-state.test.ts	Updates dynamics tests and adds coverage for “discrete fields untouched” behavior.
libs/@hashintel/petrinaut-core/src/simulation/engine/compute-next-frame.ts	Updates frame evolution to apply dynamics on packed bytes and write results back by byte offsets.
libs/@hashintel/petrinaut-core/src/simulation/engine/compute-next-frame.test.ts	Updates next-frame tests to assert on decoded tokens rather than raw numeric buffers.
libs/@hashintel/petrinaut-core/src/simulation/engine/check-transition-enablement.test.ts	Updates enablement tests for byte-offset/stride-based place state and v2 frame buffers.
libs/@hashintel/petrinaut-core/src/simulation/engine/build-simulation.ts	Updates initial marking packing and dynamics compilation to use packed token layouts and byte buffers.
libs/@hashintel/petrinaut-core/src/simulation/engine/build-simulation.test.ts	Updates build-simulation tests to validate packed layout properties and decoded token records.
libs/@hashintel/petrinaut-core/src/simulation/ARCHITECTURE.md	Updates architecture docs to describe frame format v2 and packed-struct token layout.
libs/@hashintel/petrinaut-core/src/simulation/api.ts	Removes SimulationPlaceTokenValues and getPlaceTokenValues from the public frame reader API.
libs/@hashintel/petrinaut-core/src/index.ts	Exports the new token-layout module API from the package root and removes the old token-values export.
libs/@hashintel/petrinaut-core/src/actual-mode/timeline.ts	Removes getPlaceTokenValues implementation from actual-mode timeline frame reader.
.changeset/h-6519-discrete-token-attribute-types.md	Bumps changeset impact to minor for both packages and notes the v2 frame format change.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 50 out of 51 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (2)

libs/@hashintel/petrinaut-core/src/simulation/engine/compute-possible-transition.ts:107

• Typo in comment: "acumulate" → "accumulate".

  // TODO: This should acumulate lambda over time, but for now we just consider that lambda is constant per combination.
  // (just multiply by time since last transition)

libs/@hashintel/petrinaut-core/src/simulation/engine/remove-tokens-from-simulation-frame.ts:62

• This error message still refers to "dimensions"; it should describe the non-zero strideBytes case to match the new frame format terminology.

      if (typeof indices === "number") {
        throw new Error(
          `For place ${placeId} with dimensions, expected Set of token indices to remove, got number.`,
        );
      }

[Copilot]

Pull request overview

Copilot reviewed 49 out of 49 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

libs/@hashintel/petrinaut-core/src/simulation/frames/internal-frame.ts:58

• Several header/enum field names still refer to "Value"/"Values" even though the frame format is now byte-addressed (e.g., placeValueOffsetsOffset, TokenValuesOffset). This is likely to cause confusion for future edits (especially when reasoning about units: values vs bytes). Consider renaming these to placeByteOffsetsOffset / TokenBytesOffset (and updating related locals like tokenValuesOffset) to reflect the v2 semantics.

type EngineFrameHeader = {
  placeCount: number;
  transitionCount: number;
  tokenByteLength: number;
  placeCountsOffset: number;
  placeValueOffsetsOffset: number;
  transitionElapsedOffset: number;
  transitionFiringCountsOffset: number;
  transitionFiredFlagsOffset: number;
  tokenValuesOffset: number;

[kube]

• FE-1153: Collapse per-type switches into a type-policy registry #8964
• FE-769: Add String token dimension type (string interning) #8956
• FE-1121, FE-769, FE-1153: UUID & String token dimension types, type-policy registry #8953
• FE-1139: Simulation architecture documentation #8951
• FE-1130: Token frame format v2 — packed struct token layout #8944 👈 (View in Graphite)
• FE-1129: Token encoding playground (Storybook) #8943
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[Copilot]

Pull request overview

Copilot reviewed 51 out of 51 changed files in this pull request and generated 2 comments.

The base branch was changed.