Argos collector (map_v3)

[colby-nyce]

ArgosArch.pptx

Since this is a very large PR, please review the files in the following order. If you want to review the easy files first, these are standalone utilities/changes without much Argos context:

Timestamps.hpp

• Simple wrapper around a uint64_t to add timestamps to collected data
• Unit tests use backpointer to local variable
• Sparta uses std::function for Scheduler::getCurrentTime()

StreamBuffer.hpp

• Wrapper around a byte buffer

CollectedData.hpp

• Wrapper around a StreamBuffer
• Collectable ID (CID, uint16_t) always starts off the byte buffer

Dump.hpp

• Utility to pretty print DB tables
• Controlled by --simdb-verbose in Sparta
• Used for debugging only
• Related changes: App.hpp, AppManager.hpp

Compress.hpp

• Fix zlib.decompress() bug for empty blobs
• Bug not encountered during real Argos use; suggested by AI during edge case bashing

TypeTraits.hpp

• Largely taken directly from Sparta's MetaStructs.hpp (Avi's code for pair collection)
• New code I added near the bottom: pod_convertible
• Note that has_ostream_operator / has_sparta_pair_definition_type will be changed when rest of Sparta collection code is moved to SimDB

ValidValue.hpp

• This whole class could just be identical to Sparta's
• Can we move sparta::utils::ValidValue to SimDB and just keep an alias in Sparta?

TinyStrings.hpp

• Switch to in-memory string map instead of requiring DatabaseManager
• This was needed for some Sparta unit tests that don't have a DB (such as collecting a Queue without SimDB at all)

Now getting into the core changes...

PipelineDataTypes.hpp

• Data structures used in all the DB pipeline's ConcurrentQueues in between async stages

PipelineStagerInterface.hpp

• Interface class to receive collected data, notifications, and open/close state changes

EntryPoint.hpp

• All EntryPoints are talking to the same PipelineStagerInterface
• Sparta's Collectable/IterableCollector each have their own EntryPoint

ArgosCollector.hpp

• Main entry point into the Argos collection system
• Defines schema and DB pipeline
• Writes metadata before/after simulation

Checkpointer.hpp
Checkpoint.hpp
CheckpointDeltas.hpp
CheckpointEncodings.hpp

• Implements delta encoding for scalars and containers
• Done on the 1st async pipeline stage

blob_handlers.py
blob_iterator.py

• Implements single-pass blob iteration
• Undoes delta encoding to get final data values at a specific time

Don't really need to review the rest of the python code.

test/argos/*

• Integration-level testing
• Largely written by AI
• Runs 3 fully randomized simulations at heartbeats 1, 3, and 10
• Invokes python code to compare all deserialized values for all three databases for exact match

[colbynyce-mips]

Here are the latest performance numbers compared to the legacy collector:

• 7.5% faster
• 90% smaller databases / files

Further perf improvements probably have to wait until the pair collector code from Sparta is moved to SimDB, as nothing really stands out as worth the effort right now.

[Copilot]

Pull request overview

Copilot reviewed 76 out of 76 changed files in this pull request and generated 15 comments.

[Copilot]

Pull request overview

Copilot reviewed 69 out of 69 changed files in this pull request and generated 5 comments.

[colbynyce-mips]

After more delta encodings were added in commit aa4d0be, the database dropped in size again. Previously 90% smaller than legacy, now 92.6% smaller. The actual DB sizes:

Legacy: 530 MB
Pre-aa4d0be: 52 MB
Post-aa4d0be: 39 MB

Runtime was not affected since delta encoding is done on the pipeline's background threads. The encoder thread is still sleeping 67% of the time, so we can add more compression algos later.