diff --git a/docs/docs/00100-intro/00100-getting-started/00250-zen-of-spacetimedb.md b/docs/docs/00100-intro/00100-getting-started/00250-zen-of-spacetimedb.md index bcf7da63793..eb62cbebc58 100644 --- a/docs/docs/00100-intro/00100-getting-started/00250-zen-of-spacetimedb.md +++ b/docs/docs/00100-intro/00100-getting-started/00250-zen-of-spacetimedb.md @@ -26,14 +26,18 @@ Traditional stack: SpacetimeDB: ## Everything is Persistent -You will ask, does everything need to be persistent? Won't that be a lot of data? Won't that be slow. The answers are yes, no, and no. SpacetimeDB persists everything by default, even the full history of everything that has every changed. It should be **your choice** to delete data, not the database's. SpacetimeDB gives you that choice. +SpacetimeDB persists everything by default, including the full history of any rows that have ever changed. + +You will ask, does everything need to be persistent? Won't that be a lot of data? Well, you would be surprised! For example, updating 1 million player transforms 10 times per second for a year uses roughly 10 petabytes of data, uncompressed. SpacetimeDB can compress that sort of data by about 5-10x, meaning that keeping every position for every player for a game with a million concurrent players uses only about 1-2 petabytes per year. Storing that much data in Amazon S3 would only cost you between $2,300 and $5,600 per month. A fraction of the cost of a single engineer or data scientist! + +You can of course choose to delete the historical data, but it should be **your choice** to delete data, not the database's. SpacetimeDB gives you that choice. + +Won't it be slow to persist everything? No. SpacetimeDB is designed so that persistence guarantees only ever increase latency and never decrease throughput! Modern SSDs can write upwards of 15 GB/s of data to disk. DRAM can only do about 4x more. Let's actually use that Samsung-given bandwidth. SpacetimeDB holds all your data in memory for blazing-fast access, but automatically persists everything to disk. You get the speed of in-memory computing with the durability of a traditional database. You will be tempted to ask for "ephemeral state". This is a mistake. Persistent everything allows your app to recover to the *exact* state it was in. In principle, you could even debug your production app in the state it was in in the past with a time-traveling debugger. -SpacetimeDB is designed so that persistence guarantees only ever increase latency and never decrease throughput! Modern SSDs can write upwards of 15 GB/s of data to disk. DRAM can only do about 4x more. Let's actually use that Samsung-given bandwidth. - Write your code as if memory were infinite and permanent. Insert rows freely. Query without fear. SpacetimeDB handles the persistence, you handle the logic. ## Everything is Real-Time diff --git a/docs/llms/docs-benchmark-analysis.md b/docs/llms/docs-benchmark-analysis.md index faa301f26fc..3e5887a6791 100644 --- a/docs/llms/docs-benchmark-analysis.md +++ b/docs/llms/docs-benchmark-analysis.md @@ -8,243 +8,246 @@ Generated from: `/__w/SpacetimeDB/SpacetimeDB/tools/xtask-llm-benchmark/../../do --- -## Analysis of SpacetimeDB Benchmark Test Failures - -### Rust / rustdoc_json Failures - -#### Compile/Publish Errors (2 Failures) - -##### Failure Group 1: `t_002_scheduled_table` and `t_017_scheduled_columns` -1. **The generated code**: - ```rust - use spacetimedb::{table, reducer, ReducerContext, Table, ScheduleAt}; - - #[table(name = tick_timer, schedule(reducer = tick, column = scheduled_at))] - pub struct TickTimer { - #[primary_key] - #[auto_inc] - scheduled_id: u64, - scheduled_at: ScheduleAt, - } - - #[reducer(init)] - pub fn init(ctx: &ReducerContext) { - if ctx.db.tick_timer().count() == 0 { - ctx.db.tick_timer().insert(TickTimer { - scheduled_id: 0, - scheduled_at: ScheduleAt::RepeatMicros(50_000), - }); - } - } - - #[reducer] - pub fn tick(_ctx: &ReducerContext, _row: TickTimer) { - } - ``` - -2. **The golden example**: - ```rust - use spacetimedb::{reducer, table, ReducerContext, ScheduleAt, Table}; - use std::time::Duration; - - #[table(name = tick_timer, scheduled(tick))] - pub struct TickTimer { - #[primary_key] - #[auto_inc] - pub scheduled_id: u64, - pub scheduled_at: ScheduleAt, - } - - #[reducer] - pub fn tick(_ctx: &ReducerContext, _schedule: TickTimer) { - } - - #[reducer(init)] - pub fn init(ctx: &ReducerContext) { - let every_50ms: ScheduleAt = Duration::from_millis(50).into(); - ctx.db.tick_timer().insert(TickTimer { - scheduled_id: 0, - scheduled_at: every_50ms, - }); - } - ``` - -3. **The error**: - - `publish_error: spacetime publish failed (exit=1)` - -4. **Explain the difference**: - - The generated code used `ScheduleAt::RepeatMicros(50_000)` instead of the correct `ScheduleAt::Interval(Duration::from_millis(50).into())`. The way the scheduling was set up was incorrect. - -5. **Root cause**: - - The documentation does not clearly specify the constructor syntax for `ScheduleAt` nor how to correctly set up the scheduled tasks in this context. - -6. **Recommendation**: - - Update documentation to provide examples of different constructors for `ScheduleAt`, specifically emphasizing how to define intervals correctly. +# Analysis of SpacetimeDB Benchmark Test Failures + +## Rust / Rustdoc_JSON Failures + +### Compile/Publish Errors (4 failures) + +#### Failure Group: Scheduled Functions + +**1. Code Examples** + +**LLM Output**: +```rust +#[spacetimedb::table(name = tick_timer, schedule(reducer = tick, column = scheduled_at))] +pub struct TickTimer { + #[primary_key] + #[auto_inc] + scheduled_id: u64, + scheduled_at: ScheduleAt, +} + +#[spacetimedb::reducer(init)] +pub fn init(ctx: &ReducerContext) { + ctx.db.tick_timer().insert(TickTimer { + scheduled_id: 0, + scheduled_at: ScheduleAt::Interval(Duration::from_micros(50_000).into()), + }); +} + +#[spacetimedb::reducer] +pub fn tick(_ctx: &ReducerContext, _row: TickTimer) { +} +``` + +**Expected (golden)**: +```rust +#[table(name = tick_timer, scheduled(tick))] +pub struct TickTimer { + #[primary_key] + #[auto_inc] + pub scheduled_id: u64, + pub scheduled_at: ScheduleAt, +} + +#[reducer(init)] +pub fn init(ctx: &ReducerContext) -> Result<(), String> { + ctx.db.tick_timer().insert(TickTimer { + scheduled_id: 0, + scheduled_at: ScheduleAt::Interval(Duration::from_millis(50).into()), + }); + Ok(()) +} + +#[reducer] +pub fn tick(_ctx: &ReducerContext, _row: TickTimer) -> Result<(), String> { + Ok(()) +} +``` + +**2. Error**: `publish_error: spacetime publish failed (exit=1)` + +**3. Differences**: +- The expected code uses the correct syntax for initializing the scheduled function and includes return types (`Result<(), String>`). +- The `scheduled_at` value is initialized using `Duration::from_millis(50)` instead of `Duration::from_micros(50_000)`. + +**4. Root Cause**: +The generated functions lacked return types, which are critical for reducers in Rust, and the duration initialization was incorrect. + +**5. Recommendation**: +- Update the documentation to specify that all reducer functions should have return types. +- Clarify the correct usage of time duration initialization for `ScheduleAt`. + +#### Failure Group: Structs in Tables + +**1. Code Examples** + +**LLM Output**: +```rust +#[derive(SpacetimeType)] +pub struct Position { + x: i32, + y: i32, +} + +#[table(name = entity)] +pub struct Entity { + #[primary_key] + id: i32, + pos: Position, +} +``` + +**Expected (golden)**: +```rust +#[derive(SpacetimeType, Clone, Debug)] +pub struct Position { + pub x: i32, + pub y: i32, +} + +#[table(name = entity)] +pub struct Entity { + #[primary_key] + pub id: i32, + pub pos: Position, +} +``` + +**2. Error**: `schema_parity: reducers differ - expected [], got ["add_entity()"]` + +**3. Differences**: +- The original code lacked public access modifiers for struct fields, which is required for the SpacetimeDB framework. + +**4. Root Cause**: +LLM output does not follow the requirement for public fields in data structures intended for use with SpacetimeDB. + +**5. Recommendation**: +- Update the docstring examples to include access modifiers for struct fields when using `SpacetimeType`. --- -#### Other Failures (5 failures) - -##### Failure Group 2: `t_013_spacetime_sum_type`, `t_015_product_type_columns`, `t_016_sum_type_columns`, `t_018_constraints`, `t_020_ecs` -1. **The generated code**: - ```rust - use spacetimedb::{table, reducer, ReducerContext, Table, SpacetimeType}; - - #[derive(SpacetimeType)] - pub struct Rect { - width: i32, - height: i32, - } - - #[table(name = result)] - pub struct ResultRow { - #[primary_key] - id: i32, - value: Shape, - } - - #[reducer] - pub fn set_circle(ctx: &ReducerContext, id: i32, radius: i32) { - ctx.db.result().insert(ResultRow { - id, - value: Shape::Circle(radius), - }); - } - ``` - -2. **The golden example**: - ```rust - use spacetimedb::{reducer, table, ReducerContext, SpacetimeType, Table}; - - #[derive(SpacetimeType, Clone, Debug)] - pub struct Rect { - pub width: i32, - pub height: i32, - } - - #[table(name = result)] - pub struct ResultRow { - #[primary_key] - pub id: i32, - pub value: Shape, - } - - #[reducer] - pub fn set_circle(ctx: &ReducerContext, id: i32, radius: i32) { - ctx.db.result().insert(ResultRow { id, value: Shape::Circle(radius) }); - } - ``` - -3. **The error**: - - `spacetime sql failed: no such table: result` - - `spacetime sql failed: no such table: profile` - - `spacetime sql failed: no such table: drawings` - -4. **Explain the difference**: - - The generated code omits the `pub` visibility keyword for fields and structs, which prevents proper access by the macros that generate the expected database schema. Additionally, the enum `Shape` wasn't declared correctly in the generated code. - -5. **Root cause**: - - Lack of proper visibility (missing `pub`) for structs and enum fields was not clearly emphasized in the documentation, leading to access issues. - -6. **Recommendation**: - - Provide clear guidelines in the documentation regarding the necessity of using `pub` for struct and enum fields when working with SpacetimeDB components. Include example schemas with visibility marked. +### Other Failures (4 failures) ---- +#### Failure Group: Incomplete Functions and Missing Return Types + +**1. Code Examples** + +**LLM Output**: +```rust +#[spacetimedb::reducer] +pub fn empty_reducer_no_args(_ctx: &ReducerContext) { } + +#[spacetimedb::reducer] +pub fn empty_reducer_with_int(_ctx: &ReducerContext, count: i32) { } +``` + +**Expected (golden)**: +```rust +#[reducer] +pub fn empty_reducer_no_args(ctx: &ReducerContext) -> Result<(), String> { + Ok(()) +} + +#[reducer] +pub fn empty_reducer_with_int(ctx: &ReducerContext, count: i32) -> Result<(), String> { + Ok(()) +} +``` + +**2. Error**: `schema_parity: describe failed` -### Rust / docs Failures (22 total) +**3. Differences**: +- The reducers in the generated code did not have the required return types (`Result<(), String>`) and failed to match the expected function signatures. -#### Timeout Issues (1 failure) +**4. Root Cause**: +Inconsistent documentation specifying both function signatures and expected return types for reducers. -1. **Failure Group**: `t_015_product_type_columns` - - **Expected**: Modify the query logic to ensure no unnecessary long-running operations exist. - - **Recommendation**: Provide timeout considerations in the documentation to ensure optimization options are explored to prevent long-running tasks. +**5. Recommendation**: +- Ensure that all documentation explicitly mentions that all reducer functions need to return `Result<(), String>`. --- -### C# / docs Failures (4 total) - -#### C# Failure Group: `t_014_elementary_columns`, `t_016_sum_type_columns`, `t_017_scheduled_columns`, `t_020_ecs` -1. **The generated code**: - ```csharp - using SpacetimeDB; - - public static partial class Module - { - [SpacetimeDB.Table(Name = "Primitive", Public = true)] - public partial struct Primitive - { - [SpacetimeDB.PrimaryKey] - public int Id; - public int Count; - public long Total; - public float Price; - public double Ratio; - public bool Active; - public string Name; - } - - [SpacetimeDB.Reducer] - public static void Seed(ReducerContext ctx) - { - ctx.Db.Primitive.Insert(new Primitive - { - Id = 1, - Count = 2, - Total = 3000000000L, - Price = 1.5f, - Ratio = 2.25, - Active = true, - Name = "Alice" - }); - } - } - ``` - -2. **The golden example**: - ```csharp - using SpacetimeDB; - - public static partial class Module - { - [Table(Name = "Primitive")] - public partial struct Primitive - { - [PrimaryKey] public int Id; - public int Count; - public long Total; - public float Price; - public double Ratio; - public bool Active; - public string Name; - } - - [Reducer] - public static void Seed(ReducerContext ctx) - { - ctx.Db.Primitive.Insert(new Primitive { - Id = 1, - Count = 2, - Total = 3000000000, - Price = 1.5f, - Ratio = 2.25, - Active = true, - Name = "Alice" - }); - } - } - ``` - -3. **The error**: `no such table: primitive` - -4. **Explain the difference**: - - Missing the `public` access modifier in the declaration of the `Table` attribute. The expected syntax properly utilizes attributes defined in the library. - -5. **Root cause**: - - Documentation may lack clarity about access modifiers, especially when it comes to how they affect visibility in entities. - -6. **Recommendation**: - - Ensure C# documentation includes explicit examples where `public` is required in class and struct declarations to prevent access issues with tables. +## C# / Docs Failures + +### Other Failures (4 failures) + +#### Failure Group: Table Not Found Errors + +**1. Code Examples** + +**LLM Output**: +```csharp +[SpacetimeDB.Table(Name = "Primitive", Public = true)] +public partial struct Primitive +{ + [SpacetimeDB.PrimaryKey] + public int Id; + public int Count; + public long Total; + public float Price; + public double Ratio; + public bool Active; + public string Name; +} + +[SpacetimeDB.Reducer] +public static void Seed(ReducerContext ctx) +{ + ctx.Db.Primitive.Insert(new Primitive + { + Id = 1, + Count = 2, + Total = 3000000000L, + Price = 1.5f, + Ratio = 2.25, + Active = true, + Name = "Alice" + }); +} +``` + +**Expected (golden)**: +```csharp +[Table(Name = "Primitive")] +public partial struct Primitive +{ + [PrimaryKey] public int Id; + public int Count; + public long Total; + public float Price; + public double Ratio; + public bool Active; + public string Name; +} + +[Reducer] +public static void Seed(ReducerContext ctx) +{ + ctx.Db.Primitive.Insert(new Primitive { + Id = 1, + Count = 2, + Total = 3000000000, + Price = 1.5f, + Ratio = 2.25, + Active = true, + Name = "Alice" + }); +} +``` + +**2. Error**: `no such table: 'primitive'` + +**3. Differences**: +- The generated code defined the table with the `Public` modifier unnecessarily and lacked proper annotations in the expected format. + +**4. Root Cause**: +Inadequate understanding of how to define database tables properly in the SpacetimeDB context from the documentation. + +**5. Recommendation**: +- Revise the documentation to emphasize how tables should be correctly defined with respect to visibility modifiers. --- -By addressing the above gaps in documentation and ensuring that generated samples adhere to the expected outcomes, we can significantly reduce the number of failures in future benchmarks. +This analysis highlights common patterns in the failures, including issues with function return types, public access modifiers, and correct syntax for API functions in both Rust and C#. The recommendations aim to enhance clarity and ensure adherence to SpacetimeDB’s requirements. diff --git a/docs/llms/docs-benchmark-comment.md b/docs/llms/docs-benchmark-comment.md index b26d4820585..4e0c404474c 100644 --- a/docs/llms/docs-benchmark-comment.md +++ b/docs/llms/docs-benchmark-comment.md @@ -2,16 +2,16 @@ | Language | Mode | Category | Tests Passed | Task Pass % | |----------|------|----------|--------------|-------------| -| Rust | rustdoc_json | basics | 26/27 | 91.7% ⬆️ +8.3% | -| Rust | rustdoc_json | schema | 23/34 | 65.3% ⬆️ +10.0% | -| Rust | rustdoc_json | **total** | 49/61 | **79.7%** ⬆️ +9.1% | +| Rust | rustdoc_json | basics | 25/27 | 83.3% | +| Rust | rustdoc_json | schema | 22/34 | 62.0% ⬆️ +6.7% | +| Rust | rustdoc_json | **total** | 47/61 | **73.6%** ⬆️ +3.0% | | Rust | docs | basics | 5/27 | 11.1% | -| Rust | docs | schema | 8/32 | 20.5% ⬆️ +8.0% | -| Rust | docs | **total** | 13/59 | **15.4%** ⬆️ +3.6% | +| Rust | docs | schema | 8/34 | 20.5% ⬆️ +8.0% | +| Rust | docs | **total** | 13/61 | **15.4%** ⬆️ +3.6% | | C# | docs | basics | 27/27 | 100.0% | | C# | docs | schema | 25/34 | 73.7% | | C# | docs | **total** | 52/61 | **88.0%** | _Compared against master branch baseline_ -Generated at: 2026-01-27T20:01:19.767Z +Generated at: 2026-01-27T22:07:18.787Z diff --git a/docs/llms/docs-benchmark-details.json b/docs/llms/docs-benchmark-details.json index 5d08c34e832..cdca7fed795 100644 --- a/docs/llms/docs-benchmark-details.json +++ b/docs/llms/docs-benchmark-details.json @@ -19,7 +19,7 @@ "model_name": "GPT-5", "total_tests": 1, "passed_tests": 1, - "llm_output": "use spacetimedb::ReducerContext;\n\n#[spacetimedb::reducer]\npub fn empty_reducer_no_args(_ctx: &ReducerContext) {\n}\n\n#[spacetimedb::reducer]\npub fn empty_reducer_with_int(_ctx: &ReducerContext, count: i32) {\n let _ = count;\n}\n\n#[spacetimedb::reducer]\npub fn empty_reducer_with_string(_ctx: &ReducerContext, name: String) {\n let _ = name;\n}\n\n#[spacetimedb::reducer]\npub fn empty_reducer_with_two_args(_ctx: &ReducerContext, count: i32, name: String) {\n let _ = (count, name);\n}\n\n#[spacetimedb::reducer]\npub fn empty_reducer_with_three_args(_ctx: &ReducerContext, active: bool, ratio: f32, label: String) {\n let _ = (active, ratio, label);\n}", + "llm_output": "use spacetimedb::ReducerContext;\n\n#[spacetimedb::reducer]\npub fn empty_reducer_no_args(_ctx: &ReducerContext) {\n}\n\n#[spacetimedb::reducer]\npub fn empty_reducer_with_int(_ctx: &ReducerContext, count: i32) {\n}\n\n#[spacetimedb::reducer]\npub fn empty_reducer_with_string(_ctx: &ReducerContext, name: String) {\n}\n\n#[spacetimedb::reducer]\npub fn empty_reducer_with_two_args(_ctx: &ReducerContext, count: i32, name: String) {\n}\n\n#[spacetimedb::reducer]\npub fn empty_reducer_with_three_args(_ctx: &ReducerContext, active: bool, ratio: f32, label: String) {\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-000-empty-reducers-golden", @@ -35,15 +35,15 @@ "llm_db": "basics-t-000-empty-reducers-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:33615", + "server": "http://127.0.0.1:45511", "tables_diff": null, "tables_equal": true } } }, "vendor": "openai", - "started_at": "2026-01-27T19:50:33.923590167Z", - "finished_at": "2026-01-27T19:51:02.622615452Z" + "started_at": "2026-01-27T21:56:30.430313727Z", + "finished_at": "2026-01-27T21:57:10.467939149Z" }, "t_001_basic_tables": { "hash": "2a0f4ec42cac03fb3da542793b311e4b399845c688bb0660a6c231c82e96c56a", @@ -69,15 +69,15 @@ "llm_db": "basics-t-001-basic-tables-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:33615", + "server": "http://127.0.0.1:45511", "tables_diff": null, "tables_equal": true } } }, "vendor": "openai", - "started_at": "2026-01-27T19:50:58.838771838Z", - "finished_at": "2026-01-27T19:51:33.707208916Z" + "started_at": "2026-01-27T21:56:53.005501208Z", + "finished_at": "2026-01-27T21:57:21.645370449Z" }, "t_002_scheduled_table": { "hash": "2a0f4ec42cac03fb3da542793b311e4b399845c688bb0660a6c231c82e96c56a", @@ -87,7 +87,7 @@ "model_name": "GPT-5", "total_tests": 1, "passed_tests": 0, - "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table, ScheduleAt};\n\n#[table(name = tick_timer, schedule(reducer = tick, column = scheduled_at))]\npub struct TickTimer {\n #[primary_key]\n #[auto_inc]\n scheduled_id: u64,\n scheduled_at: ScheduleAt,\n}\n\n#[reducer(init)]\npub fn init(ctx: &ReducerContext) {\n if ctx.db.tick_timer().count() == 0 {\n ctx.db\n .tick_timer()\n .insert(TickTimer { scheduled_id: 0, scheduled_at: ScheduleAt::repeat_micros(50_000) });\n }\n}\n\n#[reducer]\npub fn tick(_ctx: &ReducerContext) {\n}", + "llm_output": "use spacetimedb::{ReducerContext, Table, ScheduleAt};\n\n#[spacetimedb::table(name = tick_timer, schedule(reducer = tick, column = scheduled_at))]\npub struct TickTimer {\n #[primary_key]\n #[auto_inc]\n scheduled_id: u64,\n scheduled_at: ScheduleAt,\n}\n\n#[spacetimedb::reducer(init)]\npub fn init(ctx: &ReducerContext) {\n if ctx.db.tick_timer().count() == 0 {\n ctx.db\n .tick_timer()\n .insert(TickTimer {\n scheduled_id: 0,\n scheduled_at: ScheduleAt::repeat_micros(50_000),\n });\n }\n}\n\n#[spacetimedb::reducer]\npub fn tick(_ctx: &ReducerContext) {\n // Implement your periodic logic here\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-002-scheduled-table-golden", @@ -99,14 +99,14 @@ "pass": false, "partial": 0.0, "notes": { - "error": "spacetime publish failed (exit=1)\n--- stderr ---\n Updating crates.io index\n Locking 72 packages to latest compatible versions\n Adding generic-array v0.14.7 (available: v0.14.9)\n Adding spacetimedb v1.11.1 (available: v1.11.3)\n Adding spacetimedb-bindings-macro v1.11.1 (available: v1.11.3)\n Adding spacetimedb-bindings-sys v1.11.1 (available: v1.11.3)\n Adding spacetimedb-lib v1.11.1 (available: v1.11.3)\n Adding spacetimedb-primitives v1.11.1 (available: v1.11.3)\n Adding spacetimedb-sats v1.11.1 (available: v1.11.3)\n Compiling proc-macro2 v1.0.106\n Compiling quote v1.0.44\n Compiling unicode-ident v1.0.22\n Compiling typenum v1.19.0\n Compiling version_check v0.9.5\n Compiling autocfg v1.5.0\n Compiling serde_core v1.0.228\n Compiling heck v0.5.0\n Compiling cfg-if v1.0.4\n Compiling shlex v1.3.0\n Compiling either v1.15.0\n Compiling serde v1.0.228\n Compiling find-msvc-tools v0.1.8\n Compiling zerocopy v0.8.34\n Compiling thiserror v1.0.69\n Compiling bitflags v2.10.0\n Compiling nohash-hasher v0.2.0\n Compiling anyhow v1.0.100\n Compiling keccak v0.1.5\n Compiling bytes v1.11.0\n Compiling humantime v2.3.0\n Compiling convert_case v0.4.0\n Compiling zmij v1.0.17\n Compiling heck v0.4.1\n Compiling arrayvec v0.7.6\n Compiling second-stack v0.3.5\n Compiling serde_json v1.0.149\n Compiling hex v0.4.3\n Compiling bytemuck v1.24.0\n Compiling getrandom v0.2.17\n Compiling itoa v1.0.17\n Compiling smallvec v1.15.1\n Compiling arrayref v0.3.9\n Compiling itertools v0.12.1\n Compiling spacetimedb-lib v1.11.1\n Compiling constant_time_eq v0.4.2\n Compiling log v0.4.29\n Compiling memchr v2.7.6\n Compiling rand_core v0.6.4\n Compiling cc v1.2.54\n Compiling scoped-tls v1.0.1\n Compiling generic-array v0.14.7\n Compiling num-traits v0.2.19\n Compiling syn v2.0.114\n Compiling http v1.4.0\n Compiling approx v0.3.2\n Compiling chrono v0.4.43\n Compiling decorum v0.3.1\n Compiling blake3 v1.8.3\n Compiling crypto-common v0.1.7\n Compiling block-buffer v0.10.4\n Compiling digest v0.10.7\n Compiling sha3 v0.10.8\n Compiling ppv-lite86 v0.2.21\n Compiling ethnum v1.5.2\n Compiling rand_chacha v0.3.1\n Compiling rand v0.8.5\n Compiling enum-as-inner v0.6.1\n Compiling thiserror-impl v1.0.69\n Compiling derive_more v0.99.20\n Compiling spacetimedb-primitives v1.11.1\n Compiling spacetimedb-bindings-macro v1.11.1\n Compiling spacetimedb-bindings-sys v1.11.1\n Compiling spacetimedb-sats v1.11.1\n Compiling spacetimedb v1.11.1\n Compiling spacetime-module v0.1.0 (/__w/SpacetimeDB/SpacetimeDB/target/llm-runs/basics/t_002_scheduled_table/rust/server/gpt-5/llm)\nerror: expected one of: `public`, `private`, `name`, `index`, `scheduled`\n --> src/lib.rs:4:28\n |\n4 | #[table(name = tick_timer, schedule(reducer = tick, column = scheduled_at))]\n | ^^^^^^^^\n\nerror[E0422]: cannot find struct, variant or union type `TickTimer` in this scope\n --> src/lib.rs:17:21\n |\n17 | .insert(TickTimer { scheduled_id: 0, scheduled_at: ScheduleAt::repeat_micros(50_000) });\n | ^^^^^^^^^ not found in this scope\n\nerror[E0599]: no method named `tick_timer` found for struct `Local` in the current scope\n --> src/lib.rs:14:15\n |\n14 | if ctx.db.tick_timer().count() == 0 {\n | ^^^^^^^^^^ method not found in `Local`\n\nerror[E0599]: no method named `tick_timer` found for struct `Local` in the current scope\n --> src/lib.rs:16:14\n |\n15 | / ctx.db\n16 | | .tick_timer()\n | | -^^^^^^^^^^ method not found in `Local`\n | |_____________|\n |\n\nerror[E0599]: no variant or associated item named `repeat_micros` found for enum `ScheduleAt` in the current scope\n --> src/lib.rs:17:76\n |\n17 | .insert(TickTimer { scheduled_id: 0, scheduled_at: ScheduleAt::repeat_micros(50_000) });\n | ^^^^^^^^^^^^^ variant or associated item not found in `ScheduleAt`\n\nSome errors have detailed explanations: E0422, E0599.\nFor more information about an error, try `rustc --explain E0422`.\nerror: could not compile `spacetime-module` (lib) due to 5 previous errors\nError: command [\"cargo\", \"build\", \"--config=net.git-fetch-with-cli=true\", \"--target=wasm32-unknown-unknown\", \"--release\", \"--message-format=json-render-diagnostics\"] exited with code 101\n\n--- stdout ---\n", + "error": "spacetime publish failed (exit=1)\n--- stderr ---\n Updating crates.io index\n Locking 72 packages to latest compatible versions\n Adding generic-array v0.14.7 (available: v0.14.9)\n Adding spacetimedb v1.11.1 (available: v1.11.3)\n Adding spacetimedb-bindings-macro v1.11.1 (available: v1.11.3)\n Adding spacetimedb-bindings-sys v1.11.1 (available: v1.11.3)\n Adding spacetimedb-lib v1.11.1 (available: v1.11.3)\n Adding spacetimedb-primitives v1.11.1 (available: v1.11.3)\n Adding spacetimedb-sats v1.11.1 (available: v1.11.3)\n Compiling proc-macro2 v1.0.106\n Compiling unicode-ident v1.0.22\n Compiling quote v1.0.44\n Compiling version_check v0.9.5\n Compiling typenum v1.19.0\n Compiling autocfg v1.5.0\n Compiling heck v0.5.0\n Compiling serde_core v1.0.228\n Compiling cfg-if v1.0.4\n Compiling zerocopy v0.8.34\n Compiling serde v1.0.228\n Compiling either v1.15.0\n Compiling shlex v1.3.0\n Compiling find-msvc-tools v0.1.8\n Compiling bitflags v2.10.0\n Compiling thiserror v1.0.69\n Compiling nohash-hasher v0.2.0\n Compiling anyhow v1.0.100\n Compiling convert_case v0.4.0\n Compiling keccak v0.1.5\n Compiling heck v0.4.1\n Compiling arrayvec v0.7.6\n Compiling zmij v1.0.17\n Compiling bytes v1.11.0\n Compiling humantime v2.3.0\n Compiling serde_json v1.0.149\n Compiling bytemuck v1.24.0\n Compiling second-stack v0.3.5\n Compiling itoa v1.0.17\n Compiling getrandom v0.2.17\n Compiling arrayref v0.3.9\n Compiling smallvec v1.15.1\n Compiling hex v0.4.3\n Compiling constant_time_eq v0.4.2\n Compiling spacetimedb-lib v1.11.1\n Compiling itertools v0.12.1\n Compiling rand_core v0.6.4\n Compiling log v0.4.29\n Compiling memchr v2.7.6\n Compiling generic-array v0.14.7\n Compiling cc v1.2.54\n Compiling scoped-tls v1.0.1\n Compiling num-traits v0.2.19\n Compiling http v1.4.0\n Compiling syn v2.0.114\n Compiling approx v0.3.2\n Compiling chrono v0.4.43\n Compiling decorum v0.3.1\n Compiling blake3 v1.8.3\n Compiling block-buffer v0.10.4\n Compiling crypto-common v0.1.7\n Compiling digest v0.10.7\n Compiling sha3 v0.10.8\n Compiling ethnum v1.5.2\n Compiling ppv-lite86 v0.2.21\n Compiling rand_chacha v0.3.1\n Compiling rand v0.8.5\n Compiling enum-as-inner v0.6.1\n Compiling thiserror-impl v1.0.69\n Compiling derive_more v0.99.20\n Compiling spacetimedb-primitives v1.11.1\n Compiling spacetimedb-bindings-sys v1.11.1\n Compiling spacetimedb-bindings-macro v1.11.1\n Compiling spacetimedb-sats v1.11.1\n Compiling spacetimedb v1.11.1\n Compiling spacetime-module v0.1.0 (/__w/SpacetimeDB/SpacetimeDB/target/llm-runs/basics/t_002_scheduled_table/rust/server/gpt-5/llm)\nerror: expected one of: `public`, `private`, `name`, `index`, `scheduled`\n --> src/lib.rs:4:41\n |\n4 | #[spacetimedb::table(name = tick_timer, schedule(reducer = tick, column = scheduled_at))]\n | ^^^^^^^^\n\nerror[E0422]: cannot find struct, variant or union type `TickTimer` in this scope\n --> src/lib.rs:17:21\n |\n17 | .insert(TickTimer {\n | ^^^^^^^^^ not found in this scope\n\nerror[E0599]: no method named `tick_timer` found for struct `Local` in the current scope\n --> src/lib.rs:14:15\n |\n14 | if ctx.db.tick_timer().count() == 0 {\n | ^^^^^^^^^^ method not found in `Local`\n\nerror[E0599]: no method named `tick_timer` found for struct `Local` in the current scope\n --> src/lib.rs:16:14\n |\n15 | / ctx.db\n16 | | .tick_timer()\n | | -^^^^^^^^^^ method not found in `Local`\n | |_____________|\n |\n\nerror[E0599]: no variant or associated item named `repeat_micros` found for enum `ScheduleAt` in the current scope\n --> src/lib.rs:19:43\n |\n19 | scheduled_at: ScheduleAt::repeat_micros(50_000),\n | ^^^^^^^^^^^^^ variant or associated item not found in `ScheduleAt`\n\nSome errors have detailed explanations: E0422, E0599.\nFor more information about an error, try `rustc --explain E0422`.\nerror: could not compile `spacetime-module` (lib) due to 5 previous errors\nError: command [\"cargo\", \"build\", \"--config=net.git-fetch-with-cli=true\", \"--target=wasm32-unknown-unknown\", \"--release\", \"--message-format=json-render-diagnostics\"] exited with code 101\n\n--- stdout ---\n", "phase": "build_or_publish" } } }, "vendor": "openai", - "started_at": "2026-01-27T19:51:00.465423417Z", - "finished_at": "2026-01-27T19:51:47.339587032Z" + "started_at": "2026-01-27T21:57:21.645397645Z", + "finished_at": "2026-01-27T21:58:33.883597995Z" }, "t_003_struct_in_table": { "hash": "2a0f4ec42cac03fb3da542793b311e4b399845c688bb0660a6c231c82e96c56a", @@ -115,8 +115,8 @@ "golden_published": true, "model_name": "GPT-5", "total_tests": 1, - "passed_tests": 1, - "llm_output": "use spacetimedb::SpacetimeType;\n\n#[derive(SpacetimeType)]\npub struct Position {\n x: i32,\n y: i32,\n}\n\n#[spacetimedb::table(name = entity)]\npub struct Entity {\n #[primary_key]\n id: i32,\n pos: Position,\n}", + "passed_tests": 0, + "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table, SpacetimeType};\n\n#[derive(SpacetimeType)]\npub struct Position {\n x: i32,\n y: i32,\n}\n\n#[table(name = entity)]\npub struct Entity {\n #[primary_key]\n id: i32,\n pos: Position,\n}\n\n#[reducer]\npub fn add_entity(ctx: &ReducerContext, id: i32, x: i32, y: i32) {\n ctx.db.entity().insert(Entity { id, pos: Position { x, y } });\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-003-struct-in-table-golden", @@ -130,17 +130,22 @@ "notes": { "golden_db": "basics-t-003-struct-in-table-golden", "llm_db": "basics-t-003-struct-in-table-gpt-5-llm", - "reducers_diff": null, - "reducers_equal": true, - "server": "http://127.0.0.1:33615", + "reducers_diff": { + "only_golden": [], + "only_llm": [ + "add_entity()" + ] + }, + "reducers_equal": false, + "server": "http://127.0.0.1:45511", "tables_diff": null, "tables_equal": true } } }, "vendor": "openai", - "started_at": "2026-01-27T19:50:43.120116357Z", - "finished_at": "2026-01-27T19:51:34.789700080Z" + "started_at": "2026-01-27T21:57:21.911599247Z", + "finished_at": "2026-01-27T21:57:57.958186477Z" }, "t_004_insert": { "hash": "2a0f4ec42cac03fb3da542793b311e4b399845c688bb0660a6c231c82e96c56a", @@ -187,13 +192,26 @@ "llm_out": "id | name | age | active ----+---------+-----+-------- 1 | \"Alice\" | 30 | true", "query": "SELECT id, name, age, active FROM user WHERE id=1", "reducer": "insert_user", - "server": "http://127.0.0.1:33615" + "server": "http://127.0.0.1:45511" + } + }, + "schema_parity": { + "pass": true, + "partial": 1.0, + "notes": { + "golden_db": "basics-t-004-insert-golden", + "llm_db": "basics-t-004-insert-gpt-5-llm", + "reducers_diff": null, + "reducers_equal": true, + "server": "http://127.0.0.1:45511", + "tables_diff": null, + "tables_equal": true } } }, "vendor": "openai", - "started_at": "2026-01-27T19:51:00.012104606Z", - "finished_at": "2026-01-27T19:51:27.169795394Z" + "started_at": "2026-01-27T21:56:44.401349330Z", + "finished_at": "2026-01-27T21:57:14.670545355Z" }, "t_005_update": { "hash": "2a0f4ec42cac03fb3da542793b311e4b399845c688bb0660a6c231c82e96c56a", @@ -203,7 +221,7 @@ "model_name": "GPT-5", "total_tests": 3, "passed_tests": 3, - "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table};\n\n#[table(name = user)]\npub struct User {\n #[primary_key]\n id: i32,\n name: String,\n age: i32,\n active: bool,\n}\n\n#[reducer]\npub fn update_user(ctx: &ReducerContext, id: i32, name: String, age: i32, active: bool) {\n let row = User { id, name, age, active };\n ctx.db.user().id().update(row);\n}", + "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = user)]\npub struct User {\n #[primary_key]\n id: i32,\n name: String,\n age: i32,\n active: bool,\n}\n\n#[spacetimedb::reducer]\npub fn update_user(ctx: &ReducerContext, id: i32, name: String, age: i32, active: bool) {\n let table = ctx.db.user();\n if table.id().find(id).is_some() {\n let _ = table.id().update(User { id, name, age, active });\n }\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-005-update-golden", @@ -219,11 +237,18 @@ "llm_db": "basics-t-005-update-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:33615", + "server": "http://127.0.0.1:45511", "tables_diff": null, "tables_equal": true } }, + "seed_users_row": { + "pass": true, + "partial": 1.0, + "notes": { + "sql": "INSERT INTO user(id, name, age, active) VALUES (1, 'Alice', 30, true)" + } + }, "data_parity_update_user": { "pass": true, "partial": 1.0, @@ -240,20 +265,13 @@ "llm_out": "id | name | age | active ----+----------+-----+-------- 1 | \"Alice2\" | 31 | false", "query": "SELECT id, name, age, active FROM user WHERE id=1", "reducer": "update_user", - "server": "http://127.0.0.1:33615" - } - }, - "seed_users_row": { - "pass": true, - "partial": 1.0, - "notes": { - "sql": "INSERT INTO user(id, name, age, active) VALUES (1, 'Alice', 30, true)" + "server": "http://127.0.0.1:45511" } } }, "vendor": "openai", - "started_at": "2026-01-27T19:50:22.268945837Z", - "finished_at": "2026-01-27T19:51:01.660325703Z" + "started_at": "2026-01-27T21:56:26.482433771Z", + "finished_at": "2026-01-27T21:57:16.618669083Z" }, "t_006_delete": { "hash": "2a0f4ec42cac03fb3da542793b311e4b399845c688bb0660a6c231c82e96c56a", @@ -263,7 +281,7 @@ "model_name": "GPT-5", "total_tests": 3, "passed_tests": 3, - "llm_output": "use spacetimedb::{table, reducer, ReducerContext};\n\n#[table(name = user)]\npub struct User {\n #[primary_key]\n id: i32,\n name: String,\n age: i32,\n active: bool,\n}\n\n#[reducer]\npub fn delete_user(ctx: &ReducerContext, id: i32) {\n ctx.db.user().id().delete(&id);\n}", + "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table};\n\n#[table(name = user)]\npub struct User {\n #[primary_key]\n id: i32,\n name: String,\n age: i32,\n active: bool,\n}\n\n#[reducer]\npub fn delete_user(ctx: &ReducerContext, id: i32) {\n ctx.db.user().id().delete(&id);\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-006-delete-golden", @@ -286,7 +304,7 @@ "llm_db": "basics-t-006-delete-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:33615", + "server": "http://127.0.0.1:45511", "tables_diff": null, "tables_equal": true } @@ -302,8 +320,8 @@ } }, "vendor": "openai", - "started_at": "2026-01-27T19:50:18.814239252Z", - "finished_at": "2026-01-27T19:51:00.465380864Z" + "started_at": "2026-01-27T21:56:52.270086833Z", + "finished_at": "2026-01-27T21:58:00.013810045Z" }, "t_007_crud": { "hash": "2a0f4ec42cac03fb3da542793b311e4b399845c688bb0660a6c231c82e96c56a", @@ -313,7 +331,7 @@ "model_name": "GPT-5", "total_tests": 4, "passed_tests": 4, - "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table};\n\n#[table(name = user)]\npub struct User {\n #[primary_key]\n id: i32,\n name: String,\n age: i32,\n active: bool,\n}\n\n#[reducer]\npub fn crud(ctx: &ReducerContext) {\n let users = ctx.db.user();\n\n // insert (id=1, name=\"Alice\", age=30, active=true)\n users.insert(User {\n id: 1,\n name: \"Alice\".to_string(),\n age: 30,\n active: true,\n });\n\n // insert (id=2, name=\"Bob\", age=22, active=false)\n users.insert(User {\n id: 2,\n name: \"Bob\".to_string(),\n age: 22,\n active: false,\n });\n\n // update (id=1, name=\"Alice2\", age=31, active=false)\n users.id().update(User {\n id: 1,\n name: \"Alice2\".to_string(),\n age: 31,\n active: false,\n });\n\n // delete id=2\n users.id().delete(&2);\n}", + "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = user)]\npub struct User {\n #[primary_key]\n id: i32,\n name: String,\n age: i32,\n active: bool,\n}\n\n#[spacetimedb::reducer]\npub fn crud(ctx: &ReducerContext) {\n let _ = ctx.db.user().insert(User { id: 1, name: \"Alice\".to_string(), age: 30, active: true });\n let _ = ctx.db.user().insert(User { id: 2, name: \"Bob\".to_string(), age: 22, active: false });\n let updated = User { id: 1, name: \"Alice2\".to_string(), age: 31, active: false };\n let _ = ctx.db.user().id().update(updated);\n ctx.db.user().id().delete(&2);\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-007-crud-golden", @@ -321,26 +339,18 @@ "work_dir_golden": "target/llm-runs/basics/t_007_crud/rust/server/golden", "work_dir_llm": "target/llm-runs/basics/t_007_crud/rust/server/gpt-5/llm", "scorer_details": { - "crud_row_id2_deleted": { - "pass": true, - "partial": 1.0, - "notes": { - "actual": 0, - "expected": 0, - "sql": "SELECT COUNT(*) AS n FROM user WHERE id=2" - } - }, - "schema_parity": { + "crud_row_id1_parity": { "pass": true, "partial": 1.0, "notes": { + "args": [], "golden_db": "basics-t-007-crud-golden", + "golden_out": "id | name | age | active ----+----------+-----+-------- 1 | \"Alice2\" | 31 | false", "llm_db": "basics-t-007-crud-gpt-5-llm", - "reducers_diff": null, - "reducers_equal": true, - "server": "http://127.0.0.1:33615", - "tables_diff": null, - "tables_equal": true + "llm_out": "id | name | age | active ----+----------+-----+-------- 1 | \"Alice2\" | 31 | false", + "query": "SELECT id, name, age, active FROM user WHERE id=1", + "reducer": "crud", + "server": "http://127.0.0.1:45511" } }, "crud_total_count_one": { @@ -352,24 +362,32 @@ "sql": "SELECT COUNT(*) AS n FROM user" } }, - "crud_row_id1_parity": { + "crud_row_id2_deleted": { + "pass": true, + "partial": 1.0, + "notes": { + "actual": 0, + "expected": 0, + "sql": "SELECT COUNT(*) AS n FROM user WHERE id=2" + } + }, + "schema_parity": { "pass": true, "partial": 1.0, "notes": { - "args": [], "golden_db": "basics-t-007-crud-golden", - "golden_out": "id | name | age | active ----+----------+-----+-------- 1 | \"Alice2\" | 31 | false", "llm_db": "basics-t-007-crud-gpt-5-llm", - "llm_out": "id | name | age | active ----+----------+-----+-------- 1 | \"Alice2\" | 31 | false", - "query": "SELECT id, name, age, active FROM user WHERE id=1", - "reducer": "crud", - "server": "http://127.0.0.1:33615" + "reducers_diff": null, + "reducers_equal": true, + "server": "http://127.0.0.1:45511", + "tables_diff": null, + "tables_equal": true } } }, "vendor": "openai", - "started_at": "2026-01-27T19:50:19.572189963Z", - "finished_at": "2026-01-27T19:51:00.012056746Z" + "started_at": "2026-01-27T21:57:16.618713310Z", + "finished_at": "2026-01-27T21:57:59.328188811Z" }, "t_008_index_lookup": { "hash": "2a0f4ec42cac03fb3da542793b311e4b399845c688bb0660a6c231c82e96c56a", @@ -379,7 +397,7 @@ "model_name": "GPT-5", "total_tests": 3, "passed_tests": 3, - "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table};\n\n#[table(name = user)]\npub struct User {\n #[primary_key]\n id: i32,\n name: String,\n age: i32,\n active: bool,\n}\n\n#[table(name = result)]\npub struct ResultRow {\n #[primary_key]\n id: i32,\n name: String,\n}\n\n#[reducer]\npub fn lookup_user_name(ctx: &ReducerContext, id: i32) {\n if let Some(user) = ctx.db.user().id().find(id) {\n let _ = ctx.db.result().try_insert(ResultRow {\n id: user.id,\n name: user.name.clone(),\n });\n }\n}", + "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table};\n\n#[table(name = user)]\npub struct User {\n #[primary_key]\n id: i32,\n name: String,\n age: i32,\n active: bool,\n}\n\n#[table(name = result)]\npub struct ResultRow {\n #[primary_key]\n id: i32,\n name: String,\n}\n\n#[reducer]\npub fn lookup_user_name(ctx: &ReducerContext, id: i32) {\n if let Some(u) = ctx.db.user().id().find(id) {\n ctx.db.result().insert(ResultRow { id: u.id, name: u.name });\n }\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-008-index-lookup-golden", @@ -402,7 +420,7 @@ "llm_db": "basics-t-008-index-lookup-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:33615", + "server": "http://127.0.0.1:45511", "tables_diff": null, "tables_equal": true } @@ -420,13 +438,13 @@ "llm_out": "id | name ----+--------- 1 | \"Alice\"", "query": "SELECT id, name FROM result WHERE id=1", "reducer": "lookup_user_name", - "server": "http://127.0.0.1:33615" + "server": "http://127.0.0.1:45511" } } }, "vendor": "openai", - "started_at": "2026-01-27T19:50:59.032238062Z", - "finished_at": "2026-01-27T19:51:34.520084103Z" + "started_at": "2026-01-27T21:57:14.670616741Z", + "finished_at": "2026-01-27T21:58:00.233395415Z" }, "t_009_init": { "hash": "2a0f4ec42cac03fb3da542793b311e4b399845c688bb0660a6c231c82e96c56a", @@ -436,7 +454,7 @@ "model_name": "GPT-5", "total_tests": 4, "passed_tests": 4, - "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = user)]\npub struct User {\n #[primary_key]\n id: i32,\n name: String,\n age: i32,\n active: bool,\n}\n\n#[spacetimedb::reducer(init)]\npub fn init(ctx: &ReducerContext) {\n ctx.db.user().insert(User {\n id: 1,\n name: \"Alice\".to_string(),\n age: 30,\n active: true,\n });\n ctx.db.user().insert(User {\n id: 2,\n name: \"Bob\".to_string(),\n age: 22,\n active: false,\n });\n}", + "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table};\n\n#[table(name = user)]\npub struct User {\n #[primary_key]\n id: i32,\n name: String,\n age: i32,\n active: bool,\n}\n\n#[reducer(init)]\npub fn init(ctx: &ReducerContext) {\n let user = ctx.db.user();\n user.insert(User { id: 1, name: \"Alice\".to_string(), age: 30, active: true });\n user.insert(User { id: 2, name: \"Bob\".to_string(), age: 22, active: false });\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-009-init-golden", @@ -444,13 +462,13 @@ "work_dir_golden": "target/llm-runs/basics/t_009_init/rust/server/golden", "work_dir_llm": "target/llm-runs/basics/t_009_init/rust/server/gpt-5/llm", "scorer_details": { - "init_seed_bob": { + "init_total_two": { "pass": true, "partial": 1.0, "notes": { - "actual": 1, - "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM user WHERE id=2 AND name='Bob' AND age=22 AND active=false" + "actual": 2, + "expected": 2, + "sql": "SELECT COUNT(*) AS n FROM user" } }, "init_seed_alice": { @@ -470,24 +488,24 @@ "llm_db": "basics-t-009-init-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:33615", + "server": "http://127.0.0.1:45511", "tables_diff": null, "tables_equal": true } }, - "init_total_two": { + "init_seed_bob": { "pass": true, "partial": 1.0, "notes": { - "actual": 2, - "expected": 2, - "sql": "SELECT COUNT(*) AS n FROM user" + "actual": 1, + "expected": 1, + "sql": "SELECT COUNT(*) AS n FROM user WHERE id=2 AND name='Bob' AND age=22 AND active=false" } } }, "vendor": "openai", - "started_at": "2026-01-27T19:50:21.549783Z", - "finished_at": "2026-01-27T19:50:59.032183836Z" + "started_at": "2026-01-27T21:57:10.467974395Z", + "finished_at": "2026-01-27T21:57:46.281944467Z" }, "t_010_connect": { "hash": "2a0f4ec42cac03fb3da542793b311e4b399845c688bb0660a6c231c82e96c56a", @@ -513,15 +531,15 @@ "llm_db": "basics-t-010-connect-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:33615", + "server": "http://127.0.0.1:45511", "tables_diff": null, "tables_equal": true } } }, "vendor": "openai", - "started_at": "2026-01-27T19:50:19.565630879Z", - "finished_at": "2026-01-27T19:50:43.120012892Z" + "started_at": "2026-01-27T21:57:12.175282252Z", + "finished_at": "2026-01-27T21:57:47.463202429Z" }, "t_011_helper_function": { "hash": "2a0f4ec42cac03fb3da542793b311e4b399845c688bb0660a6c231c82e96c56a", @@ -554,16 +572,7 @@ "llm_out": "id | sum ----+----- 1 | 5", "query": "SELECT id, sum FROM result WHERE id=1", "reducer": "compute_sum", - "server": "http://127.0.0.1:33615" - } - }, - "helper_func_sum_abs": { - "pass": true, - "partial": 1.0, - "notes": { - "actual": 1, - "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM result WHERE id=1 AND sum=5" + "server": "http://127.0.0.1:45511" } }, "schema_parity": { @@ -574,15 +583,24 @@ "llm_db": "basics-t-011-helper-function-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:33615", + "server": "http://127.0.0.1:45511", "tables_diff": null, "tables_equal": true } + }, + "helper_func_sum_abs": { + "pass": true, + "partial": 1.0, + "notes": { + "actual": 1, + "expected": 1, + "sql": "SELECT COUNT(*) AS n FROM result WHERE id=1 AND sum=5" + } } }, "vendor": "openai", - "started_at": "2026-01-27T19:51:00.227420335Z", - "finished_at": "2026-01-27T19:51:29.538171947Z" + "started_at": "2026-01-27T21:56:53.341032329Z", + "finished_at": "2026-01-27T21:57:21.911571496Z" }, "t_012_spacetime_product_type": { "hash": "2a0f4ec42cac03fb3da542793b311e4b399845c688bb0660a6c231c82e96c56a", @@ -591,8 +609,8 @@ "golden_published": true, "model_name": "GPT-5", "total_tests": 3, - "passed_tests": 3, - "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table, SpacetimeType};\n\n#[derive(SpacetimeType)]\npub struct Score {\n left: i32,\n right: i32,\n}\n\n#[table(name = result)]\npub struct ResultRow {\n #[primary_key]\n id: i32,\n value: Score,\n}\n\n#[reducer]\nfn set_score(ctx: &ReducerContext, id: i32, left: i32, right: i32) {\n ctx.db.result().insert(ResultRow {\n id,\n value: Score { left, right },\n });\n}", + "passed_tests": 2, + "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table, SpacetimeType};\n\n#[derive(SpacetimeType)]\npub struct Score {\n left: i32,\n right: i32,\n}\n\n#[table(name = result)]\npub struct ResultRow {\n #[primary_key]\n id: i32,\n value: Score,\n}\n\n#[reducer]\npub fn set_score(ctx: &ReducerContext, id: i32, left: i32, right: i32) {\n ctx.db.result().insert(ResultRow {\n id,\n value: Score { left, right },\n });\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-012-spacetime-product-type-golden", @@ -609,6 +627,14 @@ "sql": "SELECT COUNT(*) AS n FROM result WHERE id=1" } }, + "product_type_row_parity": { + "pass": false, + "partial": 0.0, + "notes": { + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `result`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:45511/v1/database/c20066de900672af903bd323213825ab123d970ed990ba3708988e33f63407f2/sql)\n", + "phase": "sql_golden" + } + }, "schema_parity": { "pass": true, "partial": 1.0, @@ -617,7 +643,7 @@ "llm_db": "schema-t-012-spacetime-product-type-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:33615", + "server": "http://127.0.0.1:45511", "tables_diff": null, "tables_equal": true } @@ -642,8 +668,8 @@ } }, "vendor": "openai", - "started_at": "2026-01-27T19:50:17.029809591Z", - "finished_at": "2026-01-27T19:50:58.838714798Z" + "started_at": "2026-01-27T21:55:50.881200151Z", + "finished_at": "2026-01-27T21:56:26.171932993Z" }, "t_013_spacetime_sum_type": { "hash": "2a0f4ec42cac03fb3da542793b311e4b399845c688bb0660a6c231c82e96c56a", @@ -653,7 +679,7 @@ "model_name": "GPT-5", "total_tests": 3, "passed_tests": 2, - "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table, SpacetimeType};\n\n#[derive(SpacetimeType)]\npub struct Rect {\n width: i32,\n height: i32,\n}\n\n#[derive(SpacetimeType)]\npub enum Shape {\n Circle(i32),\n Rectangle(Rect),\n}\n\n#[table(name = result)]\npub struct ResultRow {\n #[primary_key]\n id: i32,\n value: Shape,\n}\n\n#[reducer]\npub fn set_circle(ctx: &ReducerContext, id: i32, radius: i32) {\n ctx.db.result().insert(ResultRow {\n id,\n value: Shape::Circle(radius),\n });\n}", + "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table, SpacetimeType};\n\n#[derive(SpacetimeType)]\npub struct Rect {\n width: i32,\n height: i32,\n}\n\n#[derive(SpacetimeType)]\npub enum Shape {\n Circle(i32),\n Rectangle(Rect),\n}\n\n#[table(name = result)]\npub struct ResultRow {\n #[primary_key]\n id: i32,\n value: Shape,\n}\n\n#[reducer]\npub fn set_circle(ctx: &ReducerContext, id: i32, radius: i32) {\n ctx.db.result().id().delete(&id);\n ctx.db.result().insert(ResultRow { id, value: Shape::Circle(radius) });\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-013-spacetime-sum-type-golden", @@ -677,7 +703,7 @@ "llm_db": "schema-t-013-spacetime-sum-type-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:33615", + "server": "http://127.0.0.1:45511", "tables_diff": null, "tables_equal": true } @@ -690,11 +716,19 @@ "expected": 1, "sql": "SELECT COUNT(*) AS n FROM result WHERE id=1" } + }, + "sum_type_row_parity": { + "pass": false, + "partial": 0.0, + "notes": { + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `result`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:45511/v1/database/c200e5fbecfa12b09aedee1725f81fee4cc7816888145963c4128b275957d1f9/sql)\n", + "phase": "sql_golden" + } } }, "vendor": "openai", - "started_at": "2026-01-27T19:49:28.665478058Z", - "finished_at": "2026-01-27T19:50:21.549671530Z" + "started_at": "2026-01-27T21:55:50.881434595Z", + "finished_at": "2026-01-27T21:56:44.401279988Z" }, "t_014_elementary_columns": { "hash": "2a0f4ec42cac03fb3da542793b311e4b399845c688bb0660a6c231c82e96c56a", @@ -712,18 +746,13 @@ "work_dir_golden": "target/llm-runs/schema/t_014_elementary_columns/rust/server/golden", "work_dir_llm": "target/llm-runs/schema/t_014_elementary_columns/rust/server/gpt-5/llm", "scorer_details": { - "elementary_columns_row_parity": { + "elementary_columns_row_count": { "pass": true, "partial": 1.0, "notes": { - "args": [], - "golden_db": "schema-t-014-elementary-columns-golden", - "golden_out": "id | count | total | price | ratio | active | name ----+-------+------------+-------+-------+--------+--------- 1 | 2 | 3000000000 | 1.5 | 2.25 | true | \"Alice\"", - "llm_db": "schema-t-014-elementary-columns-gpt-5-llm", - "llm_out": "id | count | total | price | ratio | active | name ----+-------+------------+-------+-------+--------+--------- 1 | 2 | 3000000000 | 1.5 | 2.25 | true | \"Alice\"", - "query": "SELECT id, count, total, price, ratio, active, name FROM primitive WHERE id=1", - "reducer": "seed", - "server": "http://127.0.0.1:33615" + "actual": 1, + "expected": 1, + "sql": "SELECT COUNT(*) AS n FROM primitive WHERE id=1" } }, "schema_parity": { @@ -734,24 +763,29 @@ "llm_db": "schema-t-014-elementary-columns-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:33615", + "server": "http://127.0.0.1:45511", "tables_diff": null, "tables_equal": true } }, - "elementary_columns_row_count": { + "elementary_columns_row_parity": { "pass": true, "partial": 1.0, "notes": { - "actual": 1, - "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM primitive WHERE id=1" + "args": [], + "golden_db": "schema-t-014-elementary-columns-golden", + "golden_out": "id | count | total | price | ratio | active | name ----+-------+------------+-------+-------+--------+--------- 1 | 2 | 3000000000 | 1.5 | 2.25 | true | \"Alice\"", + "llm_db": "schema-t-014-elementary-columns-gpt-5-llm", + "llm_out": "id | count | total | price | ratio | active | name ----+-------+------------+-------+-------+--------+--------- 1 | 2 | 3000000000 | 1.5 | 2.25 | true | \"Alice\"", + "query": "SELECT id, count, total, price, ratio, active, name FROM primitive WHERE id=1", + "reducer": "seed", + "server": "http://127.0.0.1:45511" } } }, "vendor": "openai", - "started_at": "2026-01-27T19:50:12.818589409Z", - "finished_at": "2026-01-27T19:51:00.227383147Z" + "started_at": "2026-01-27T21:56:16.721976906Z", + "finished_at": "2026-01-27T21:56:53.005468623Z" }, "t_015_product_type_columns": { "hash": "2a0f4ec42cac03fb3da542793b311e4b399845c688bb0660a6c231c82e96c56a", @@ -761,7 +795,7 @@ "model_name": "GPT-5", "total_tests": 3, "passed_tests": 2, - "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table, SpacetimeType};\n\n#[derive(SpacetimeType, Clone)]\nstruct Address {\n street: String,\n zip: i32,\n}\n\n#[derive(SpacetimeType, Clone)]\nstruct Position {\n x: i32,\n y: i32,\n}\n\n#[table(name = profile)]\nstruct Profile {\n #[primary_key]\n id: i32,\n home: Address,\n work: Address,\n pos: Position,\n}\n\n#[reducer]\nfn seed(ctx: &ReducerContext) {\n let tbl = ctx.db.profile();\n if tbl.count() == 0 {\n let home = Address { street: \"1 Main\".to_string(), zip: 11111 };\n let work = Address { street: \"2 Broad\".to_string(), zip: 22222 };\n let pos = Position { x: 7, y: 9 };\n tbl.insert(Profile { id: 1, home, work, pos });\n }\n}", + "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table, SpacetimeType};\n\n#[derive(SpacetimeType)]\npub struct Address {\n street: String,\n zip: i32,\n}\n\n#[derive(SpacetimeType)]\npub struct Position {\n x: i32,\n y: i32,\n}\n\n#[table(name = profile)]\npub struct Profile {\n #[primary_key]\n id: i32,\n home: Address,\n work: Address,\n pos: Position,\n}\n\n#[reducer]\npub fn seed(ctx: &ReducerContext) {\n let home = Address { street: \"1 Main\".to_string(), zip: 11111 };\n let work = Address { street: \"2 Broad\".to_string(), zip: 22222 };\n let pos = Position { x: 7, y: 9 };\n ctx.db.profile().insert(Profile { id: 1, home, work, pos });\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-015-product-type-columns-golden", @@ -782,7 +816,7 @@ "pass": false, "partial": 0.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `profile`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:33615/v1/database/c200171a9841d8ed2a9c762ac4e081fe93ed52fe5c9e32ae9bbd874bb7e1a8b5/sql)\n", + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `profile`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:45511/v1/database/c200ccf4f06f41761313705e411551f93d64b4aff2922f89a4c80c47c5e7316c/sql)\n", "phase": "sql_golden" } }, @@ -794,15 +828,15 @@ "llm_db": "schema-t-015-product-type-columns-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:33615", + "server": "http://127.0.0.1:45511", "tables_diff": null, "tables_equal": true } } }, "vendor": "openai", - "started_at": "2026-01-27T19:49:28.666090278Z", - "finished_at": "2026-01-27T19:50:19.565536981Z" + "started_at": "2026-01-27T21:55:50.882135186Z", + "finished_at": "2026-01-27T21:56:26.482395588Z" }, "t_016_sum_type_columns": { "hash": "2a0f4ec42cac03fb3da542793b311e4b399845c688bb0660a6c231c82e96c56a", @@ -812,7 +846,7 @@ "model_name": "GPT-5", "total_tests": 3, "passed_tests": 1, - "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table, SpacetimeType};\n\n#[derive(SpacetimeType)]\npub struct Rect {\n width: i32,\n height: i32,\n}\n\n#[derive(SpacetimeType)]\npub enum Shape {\n Circle(i32),\n Rectangle(Rect),\n}\n\n#[table(name = drawing)]\npub struct Drawing {\n #[primary_key]\n id: i32,\n a: Shape,\n b: Shape,\n}\n\n#[reducer]\npub fn seed(ctx: &ReducerContext) {\n if ctx.db.drawing().id().find(1i32).is_none() {\n ctx.db.drawing().insert(Drawing {\n id: 1,\n a: Shape::Circle(10),\n b: Shape::Rectangle(Rect { width: 4, height: 6 }),\n });\n }\n}", + "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table, SpacetimeType};\n\n#[derive(SpacetimeType)]\npub struct Rect {\n width: i32,\n height: i32,\n}\n\n#[derive(SpacetimeType)]\npub enum Shape {\n Circle(i32),\n Rectangle(Rect),\n}\n\n#[table(name = drawing)]\npub struct Drawing {\n #[primary_key]\n id: i32,\n a: Shape,\n b: Shape,\n}\n\n#[reducer]\npub fn seed(ctx: &ReducerContext) {\n ctx.db.drawing().insert(Drawing {\n id: 1,\n a: Shape::Circle(10),\n b: Shape::Rectangle(Rect { width: 4, height: 6 }),\n });\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-016-sum-type-columns-golden", @@ -824,7 +858,7 @@ "pass": false, "partial": 0.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `drawings`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:33615/v1/database/c200881b5220df4b060ef4c2656171f2f8326cecff55db6c3c11d3ebbd2f5cec/sql)\n", + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `drawings`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:45511/v1/database/c2005cb32f9e8c92d208ab76edf6fb54138728597e085ad576ab57cdd18ad47e/sql)\n", "phase": "sql_golden" } }, @@ -836,7 +870,7 @@ "llm_db": "schema-t-016-sum-type-columns-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:33615", + "server": "http://127.0.0.1:45511", "tables_diff": null, "tables_equal": true } @@ -845,14 +879,14 @@ "pass": false, "partial": 0.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `drawings`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:33615/v1/database/c20074c3ee57884fdd34989755bdcfa11644a8f79ecbbdbf8f5182f2be46b5ec/sql)\n", + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `drawings`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:45511/v1/database/c20037932e455399a735ccb0f4441a8f89c3046647a5b00c855c2802bbee0b65/sql)\n", "phase": "sql" } } }, "vendor": "openai", - "started_at": "2026-01-27T19:49:28.665270093Z", - "finished_at": "2026-01-27T19:50:17.767945290Z" + "started_at": "2026-01-27T21:55:50.881577887Z", + "finished_at": "2026-01-27T21:56:53.340999329Z" }, "t_017_scheduled_columns": { "hash": "2a0f4ec42cac03fb3da542793b311e4b399845c688bb0660a6c231c82e96c56a", @@ -862,7 +896,7 @@ "model_name": "GPT-5", "total_tests": 2, "passed_tests": 0, - "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table, ScheduleAt};\n\n#[table(name = tick_timer, schedule(reducer = tick, column = scheduled_at))]\npub struct TickTimer {\n #[primary_key]\n #[auto_inc]\n scheduled_id: u64,\n scheduled_at: ScheduleAt,\n}\n\n#[reducer(init)]\npub fn init(ctx: &ReducerContext) {\n if ctx.db.tick_timer().count() == 0 {\n ctx.db.tick_timer().insert(TickTimer {\n scheduled_id: 0,\n scheduled_at: ScheduleAt::RepeatMicros(50_000),\n });\n }\n}\n\n#[reducer]\npub fn tick(_ctx: &ReducerContext, _row: TickTimer) {\n}", + "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table, ScheduleAt};\n\n#[table(name = tick_timer, schedule(reducer = tick, column = scheduled_at))]\npub struct TickTimer {\n #[primary_key]\n #[auto_inc]\n scheduled_id: u64,\n scheduled_at: ScheduleAt,\n}\n\n#[reducer(init)]\npub fn init(ctx: &ReducerContext) {\n ctx.db.tick_timer().insert(TickTimer {\n scheduled_id: 0,\n scheduled_at: ScheduleAt::RepeatEveryMicros(50_000),\n });\n}\n\n#[reducer]\npub fn tick(_ctx: &ReducerContext, _row: TickTimer) {\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-017-scheduled-columns-golden", @@ -874,14 +908,14 @@ "pass": false, "partial": 0.0, "notes": { - "error": "spacetime publish failed (exit=1)\n--- stderr ---\n Updating crates.io index\n Locking 72 packages to latest compatible versions\n Adding generic-array v0.14.7 (available: v0.14.9)\n Adding spacetimedb v1.11.1 (available: v1.11.3)\n Adding spacetimedb-bindings-macro v1.11.1 (available: v1.11.3)\n Adding spacetimedb-bindings-sys v1.11.1 (available: v1.11.3)\n Adding spacetimedb-lib v1.11.1 (available: v1.11.3)\n Adding spacetimedb-primitives v1.11.1 (available: v1.11.3)\n Adding spacetimedb-sats v1.11.1 (available: v1.11.3)\n Compiling proc-macro2 v1.0.106\n Compiling unicode-ident v1.0.22\n Compiling quote v1.0.44\n Compiling typenum v1.19.0\n Compiling version_check v0.9.5\n Compiling autocfg v1.5.0\n Compiling heck v0.5.0\n Compiling serde_core v1.0.228\n Compiling cfg-if v1.0.4\n Compiling either v1.15.0\n Compiling serde v1.0.228\n Compiling find-msvc-tools v0.1.8\n Compiling shlex v1.3.0\n Compiling zerocopy v0.8.34\n Compiling nohash-hasher v0.2.0\n Compiling anyhow v1.0.100\n Compiling thiserror v1.0.69\n Compiling bitflags v2.10.0\n Compiling keccak v0.1.5\n Compiling convert_case v0.4.0\n Compiling zmij v1.0.17\n Compiling arrayvec v0.7.6\n Compiling humantime v2.3.0\n Compiling bytes v1.11.0\n Compiling heck v0.4.1\n Compiling second-stack v0.3.5\n Compiling hex v0.4.3\n Compiling serde_json v1.0.149\n Compiling constant_time_eq v0.4.2\n Compiling itoa v1.0.17\n Compiling arrayref v0.3.9\n Compiling getrandom v0.2.17\n Compiling spacetimedb-lib v1.11.1\n Compiling bytemuck v1.24.0\n Compiling smallvec v1.15.1\n Compiling rand_core v0.6.4\n Compiling cc v1.2.54\n Compiling log v0.4.29\n Compiling memchr v2.7.6\n Compiling scoped-tls v1.0.1\n Compiling generic-array v0.14.7\n Compiling itertools v0.12.1\n Compiling num-traits v0.2.19\n Compiling http v1.4.0\n Compiling syn v2.0.114\n Compiling approx v0.3.2\n Compiling chrono v0.4.43\n Compiling decorum v0.3.1\n Compiling blake3 v1.8.3\n Compiling crypto-common v0.1.7\n Compiling block-buffer v0.10.4\n Compiling digest v0.10.7\n Compiling sha3 v0.10.8\n Compiling ethnum v1.5.2\n Compiling ppv-lite86 v0.2.21\n Compiling rand_chacha v0.3.1\n Compiling rand v0.8.5\n Compiling enum-as-inner v0.6.1\n Compiling thiserror-impl v1.0.69\n Compiling derive_more v0.99.20\n Compiling spacetimedb-primitives v1.11.1\n Compiling spacetimedb-bindings-macro v1.11.1\n Compiling spacetimedb-bindings-sys v1.11.1\n Compiling spacetimedb-sats v1.11.1\n Compiling spacetimedb v1.11.1\n Compiling spacetime-module v0.1.0 (/__w/SpacetimeDB/SpacetimeDB/target/llm-runs/schema/t_017_scheduled_columns/rust/server/gpt-5/llm)\nerror: expected one of: `public`, `private`, `name`, `index`, `scheduled`\n --> src/lib.rs:4:28\n |\n4 | #[table(name = tick_timer, schedule(reducer = tick, column = scheduled_at))]\n | ^^^^^^^^\n\nerror[E0422]: cannot find struct, variant or union type `TickTimer` in this scope\n --> src/lib.rs:15:36\n |\n15 | ctx.db.tick_timer().insert(TickTimer {\n | ^^^^^^^^^ not found in this scope\n\nerror[E0412]: cannot find type `TickTimer` in this scope\n --> src/lib.rs:23:42\n |\n23 | pub fn tick(_ctx: &ReducerContext, _row: TickTimer) {\n | ^^^^^^^^^ not found in this scope\n\nerror[E0599]: no method named `tick_timer` found for struct `Local` in the current scope\n --> src/lib.rs:14:15\n |\n14 | if ctx.db.tick_timer().count() == 0 {\n | ^^^^^^^^^^ method not found in `Local`\n\nerror[E0599]: no method named `tick_timer` found for struct `Local` in the current scope\n --> src/lib.rs:15:16\n |\n15 | ctx.db.tick_timer().insert(TickTimer {\n | ^^^^^^^^^^ method not found in `Local`\n\nerror[E0599]: no variant or associated item named `RepeatMicros` found for enum `ScheduleAt` in the current scope\n --> src/lib.rs:17:39\n |\n17 | scheduled_at: ScheduleAt::RepeatMicros(50_000),\n | ^^^^^^^^^^^^ variant or associated item not found in `ScheduleAt`\n\nerror[E0277]: invalid reducer signature\n --> src/lib.rs:23:8\n |\n 22 | #[reducer]\n | ---------- required by a bound introduced by this call\n 23 | pub fn tick(_ctx: &ReducerContext, _row: TickTimer) {\n | ^^^^ this reducer signature is not valid\n |\n = help: the trait `Reducer<'_, _>` is not implemented for fn item `for<'a> fn(&'a ReducerContext, {type error}) {tick}`\n = note: \n = note: reducer signatures must match the following pattern:\n = note: `Fn(&ReducerContext, [T1, ...]) [-> Result<(), impl Display>]`\n = note: where each `Ti` type implements `SpacetimeType`.\n = note: \nnote: required by a bound in `register_reducer`\n --> /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/spacetimedb-1.11.1/src/rt.rs:746:81\n |\n746 | pub fn register_reducer<'a, A: Args<'a>, I: FnInfo>(_: impl Reducer<'a, A>) {\n | ^^^^^^^^^^^^^^ required by this bound in `register_reducer`\n\nerror[E0277]: invalid reducer signature\n --> src/lib.rs:23:8\n |\n22 | #[reducer]\n | ---------- required by a bound introduced by this call\n23 | pub fn tick(_ctx: &ReducerContext, _row: TickTimer) {\n | ^^^^ this reducer signature is not valid\n |\n = help: the trait `Reducer<'_, _>` is not implemented for fn item `for<'a> fn(&'a ReducerContext, {type error}) {tick}`\n = note: \n = note: reducer signatures must match the following pattern:\n = note: `Fn(&ReducerContext, [T1, ...]) [-> Result<(), impl Display>]`\n = note: where each `Ti` type implements `SpacetimeType`.\n = note: \nnote: required by a bound in `invoke_reducer`\n --> /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/spacetimedb-1.11.1/src/rt.rs:45:19\n |\n44 | pub fn invoke_reducer<'a, A: Args<'a>>(\n | -------------- required by a bound in this function\n45 | reducer: impl Reducer<'a, A>,\n | ^^^^^^^^^^^^^^ required by this bound in `invoke_reducer`\n\nSome errors have detailed explanations: E0277, E0412, E0422, E0599.\nFor more information about an error, try `rustc --explain E0277`.\nerror: could not compile `spacetime-module` (lib) due to 8 previous errors\nError: command [\"cargo\", \"build\", \"--config=net.git-fetch-with-cli=true\", \"--target=wasm32-unknown-unknown\", \"--release\", \"--message-format=json-render-diagnostics\"] exited with code 101\n\n--- stdout ---\n", + "error": "spacetime publish failed (exit=1)\n--- stderr ---\n Updating crates.io index\n Locking 72 packages to latest compatible versions\n Adding generic-array v0.14.7 (available: v0.14.9)\n Adding spacetimedb v1.11.1 (available: v1.11.3)\n Adding spacetimedb-bindings-macro v1.11.1 (available: v1.11.3)\n Adding spacetimedb-bindings-sys v1.11.1 (available: v1.11.3)\n Adding spacetimedb-lib v1.11.1 (available: v1.11.3)\n Adding spacetimedb-primitives v1.11.1 (available: v1.11.3)\n Adding spacetimedb-sats v1.11.1 (available: v1.11.3)\n Compiling proc-macro2 v1.0.106\n Compiling unicode-ident v1.0.22\n Compiling quote v1.0.44\n Compiling typenum v1.19.0\n Compiling version_check v0.9.5\n Compiling autocfg v1.5.0\n Compiling heck v0.5.0\n Compiling serde_core v1.0.228\n Compiling cfg-if v1.0.4\n Compiling either v1.15.0\n Compiling shlex v1.3.0\n Compiling zerocopy v0.8.34\n Compiling serde v1.0.228\n Compiling find-msvc-tools v0.1.8\n Compiling bitflags v2.10.0\n Compiling anyhow v1.0.100\n Compiling nohash-hasher v0.2.0\n Compiling thiserror v1.0.69\n Compiling bytes v1.11.0\n Compiling heck v0.4.1\n Compiling keccak v0.1.5\n Compiling convert_case v0.4.0\n Compiling arrayvec v0.7.6\n Compiling zmij v1.0.17\n Compiling humantime v2.3.0\n Compiling smallvec v1.15.1\n Compiling itoa v1.0.17\n Compiling spacetimedb-lib v1.11.1\n Compiling second-stack v0.3.5\n Compiling getrandom v0.2.17\n Compiling arrayref v0.3.9\n Compiling bytemuck v1.24.0\n Compiling cc v1.2.54\n Compiling itertools v0.12.1\n Compiling constant_time_eq v0.4.2\n Compiling hex v0.4.3\n Compiling serde_json v1.0.149\n Compiling memchr v2.7.6\n Compiling log v0.4.29\n Compiling scoped-tls v1.0.1\n Compiling rand_core v0.6.4\n Compiling generic-array v0.14.7\n Compiling num-traits v0.2.19\n Compiling http v1.4.0\n Compiling syn v2.0.114\n Compiling blake3 v1.8.3\n Compiling approx v0.3.2\n Compiling chrono v0.4.43\n Compiling decorum v0.3.1\n Compiling block-buffer v0.10.4\n Compiling crypto-common v0.1.7\n Compiling digest v0.10.7\n Compiling sha3 v0.10.8\n Compiling ppv-lite86 v0.2.21\n Compiling rand_chacha v0.3.1\n Compiling rand v0.8.5\n Compiling ethnum v1.5.2\n Compiling enum-as-inner v0.6.1\n Compiling thiserror-impl v1.0.69\n Compiling derive_more v0.99.20\n Compiling spacetimedb-primitives v1.11.1\n Compiling spacetimedb-bindings-sys v1.11.1\n Compiling spacetimedb-bindings-macro v1.11.1\n Compiling spacetimedb-sats v1.11.1\n Compiling spacetimedb v1.11.1\n Compiling spacetime-module v0.1.0 (/__w/SpacetimeDB/SpacetimeDB/target/llm-runs/schema/t_017_scheduled_columns/rust/server/gpt-5/llm)\nerror: expected one of: `public`, `private`, `name`, `index`, `scheduled`\n --> src/lib.rs:4:28\n |\n4 | #[table(name = tick_timer, schedule(reducer = tick, column = scheduled_at))]\n | ^^^^^^^^\n\nerror[E0422]: cannot find struct, variant or union type `TickTimer` in this scope\n --> src/lib.rs:14:32\n |\n14 | ctx.db.tick_timer().insert(TickTimer {\n | ^^^^^^^^^ not found in this scope\n\nerror[E0412]: cannot find type `TickTimer` in this scope\n --> src/lib.rs:21:42\n |\n21 | pub fn tick(_ctx: &ReducerContext, _row: TickTimer) {\n | ^^^^^^^^^ not found in this scope\n\nerror[E0599]: no method named `tick_timer` found for struct `Local` in the current scope\n --> src/lib.rs:14:12\n |\n14 | ctx.db.tick_timer().insert(TickTimer {\n | ^^^^^^^^^^ method not found in `Local`\n\nerror[E0599]: no variant or associated item named `RepeatEveryMicros` found for enum `ScheduleAt` in the current scope\n --> src/lib.rs:16:35\n |\n16 | scheduled_at: ScheduleAt::RepeatEveryMicros(50_000),\n | ^^^^^^^^^^^^^^^^^ variant or associated item not found in `ScheduleAt`\n\nerror[E0277]: invalid reducer signature\n --> src/lib.rs:21:8\n |\n 20 | #[reducer]\n | ---------- required by a bound introduced by this call\n 21 | pub fn tick(_ctx: &ReducerContext, _row: TickTimer) {\n | ^^^^ this reducer signature is not valid\n |\n = help: the trait `Reducer<'_, _>` is not implemented for fn item `for<'a> fn(&'a ReducerContext, {type error}) {tick}`\n = note: \n = note: reducer signatures must match the following pattern:\n = note: `Fn(&ReducerContext, [T1, ...]) [-> Result<(), impl Display>]`\n = note: where each `Ti` type implements `SpacetimeType`.\n = note: \nnote: required by a bound in `register_reducer`\n --> /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/spacetimedb-1.11.1/src/rt.rs:746:81\n |\n746 | pub fn register_reducer<'a, A: Args<'a>, I: FnInfo>(_: impl Reducer<'a, A>) {\n | ^^^^^^^^^^^^^^ required by this bound in `register_reducer`\n\nerror[E0277]: invalid reducer signature\n --> src/lib.rs:21:8\n |\n20 | #[reducer]\n | ---------- required by a bound introduced by this call\n21 | pub fn tick(_ctx: &ReducerContext, _row: TickTimer) {\n | ^^^^ this reducer signature is not valid\n |\n = help: the trait `Reducer<'_, _>` is not implemented for fn item `for<'a> fn(&'a ReducerContext, {type error}) {tick}`\n = note: \n = note: reducer signatures must match the following pattern:\n = note: `Fn(&ReducerContext, [T1, ...]) [-> Result<(), impl Display>]`\n = note: where each `Ti` type implements `SpacetimeType`.\n = note: \nnote: required by a bound in `invoke_reducer`\n --> /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/spacetimedb-1.11.1/src/rt.rs:45:19\n |\n44 | pub fn invoke_reducer<'a, A: Args<'a>>(\n | -------------- required by a bound in this function\n45 | reducer: impl Reducer<'a, A>,\n | ^^^^^^^^^^^^^^ required by this bound in `invoke_reducer`\n\nSome errors have detailed explanations: E0277, E0412, E0422, E0599.\nFor more information about an error, try `rustc --explain E0277`.\nerror: could not compile `spacetime-module` (lib) due to 7 previous errors\nError: command [\"cargo\", \"build\", \"--config=net.git-fetch-with-cli=true\", \"--target=wasm32-unknown-unknown\", \"--release\", \"--message-format=json-render-diagnostics\"] exited with code 101\n\n--- stdout ---\n", "phase": "build_or_publish" } } }, "vendor": "openai", - "started_at": "2026-01-27T19:49:28.664841535Z", - "finished_at": "2026-01-27T19:50:33.923572352Z" + "started_at": "2026-01-27T21:55:50.882262207Z", + "finished_at": "2026-01-27T21:57:12.175265404Z" }, "t_018_constraints": { "hash": "2a0f4ec42cac03fb3da542793b311e4b399845c688bb0660a6c231c82e96c56a", @@ -891,7 +925,7 @@ "model_name": "GPT-5", "total_tests": 3, "passed_tests": 2, - "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = account, index(name = by_name, btree(columns = [name])))]\npub struct Account {\n #[primary_key]\n id: i32,\n #[unique]\n email: String,\n name: String,\n}\n\n#[spacetimedb::reducer]\npub fn seed(ctx: &ReducerContext) {\n let accounts = ctx.db.account();\n let _ = accounts.try_insert(Account { id: 1, email: \"a@example.com\".to_string(), name: \"Alice\".to_string() });\n let _ = accounts.try_insert(Account { id: 2, email: \"b@example.com\".to_string(), name: \"Bob\".to_string() });\n}", + "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table};\n\n#[table(name = account, index(name = by_name, btree(columns = [name])))]\npub struct Account {\n #[primary_key]\n id: i32,\n #[unique]\n email: String,\n name: String,\n}\n\n#[reducer]\npub fn seed(ctx: &ReducerContext) {\n ctx.db\n .account()\n .insert(Account {\n id: 1,\n email: \"a@example.com\".to_string(),\n name: \"Alice\".to_string(),\n });\n ctx.db\n .account()\n .insert(Account {\n id: 2,\n email: \"b@example.com\".to_string(),\n name: \"Bob\".to_string(),\n });\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-018-constraints-golden", @@ -899,17 +933,12 @@ "work_dir_golden": "target/llm-runs/schema/t_018_constraints/rust/server/golden", "work_dir_llm": "target/llm-runs/schema/t_018_constraints/rust/server/gpt-5/llm", "scorer_details": { - "schema_parity": { - "pass": true, - "partial": 1.0, + "constraints_row_parity_after_seed": { + "pass": false, + "partial": 0.0, "notes": { - "golden_db": "schema-t-018-constraints-golden", - "llm_db": "schema-t-018-constraints-gpt-5-llm", - "reducers_diff": null, - "reducers_equal": true, - "server": "http://127.0.0.1:33615", - "tables_diff": null, - "tables_equal": true + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `account`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:45511/v1/database/c200ff6e2bd0570e541deaad776ea03f5b0bf8dc103f206aa00417919d001d2b/sql)\n", + "phase": "sql_golden" } }, "constraints_seed_two_rows": { @@ -921,18 +950,23 @@ "sql": "SELECT COUNT(*) AS n FROM account WHERE id=2" } }, - "constraints_row_parity_after_seed": { - "pass": false, - "partial": 0.0, + "schema_parity": { + "pass": true, + "partial": 1.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `account`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:33615/v1/database/c2002f3e10afac17bd0350f8d8feb90d1ae94d73c7c44c6824e7324679a8e5b9/sql)\n", - "phase": "sql_golden" + "golden_db": "schema-t-018-constraints-golden", + "llm_db": "schema-t-018-constraints-gpt-5-llm", + "reducers_diff": null, + "reducers_equal": true, + "server": "http://127.0.0.1:45511", + "tables_diff": null, + "tables_equal": true } } }, "vendor": "openai", - "started_at": "2026-01-27T19:49:28.664543506Z", - "finished_at": "2026-01-27T19:50:17.029704757Z" + "started_at": "2026-01-27T21:55:50.881725580Z", + "finished_at": "2026-01-27T21:56:16.721950170Z" }, "t_019_many_to_many": { "hash": "2a0f4ec42cac03fb3da542793b311e4b399845c688bb0660a6c231c82e96c56a", @@ -942,7 +976,7 @@ "model_name": "GPT-5", "total_tests": 5, "passed_tests": 5, - "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = user)]\npub struct User {\n #[primary_key]\n user_id: i32,\n name: String,\n}\n\n#[spacetimedb::table(name = group)]\npub struct Group {\n #[primary_key]\n group_id: i32,\n title: String,\n}\n\n#[spacetimedb::table(\n name = membership,\n index(name = by_user, btree(columns = [user_id])),\n index(name = by_group, btree(columns = [group_id]))\n)]\npub struct Membership {\n #[primary_key]\n id: i32,\n user_id: i32,\n group_id: i32,\n}\n\n#[spacetimedb::reducer]\npub fn seed(ctx: &ReducerContext) {\n // Clear existing rows to ensure the database contains exactly the specified rows.\n let membership_ids: Vec = ctx.db.membership().iter().map(|m| m.id).collect();\n for id in membership_ids {\n ctx.db.membership().id().delete(&id);\n }\n\n let user_ids: Vec = ctx.db.user().iter().map(|u| u.user_id).collect();\n for id in user_ids {\n ctx.db.user().user_id().delete(&id);\n }\n\n let group_ids: Vec = ctx.db.group().iter().map(|g| g.group_id).collect();\n for id in group_ids {\n ctx.db.group().group_id().delete(&id);\n }\n\n // Insert users\n ctx.db.user().insert(User { user_id: 1, name: \"Alice\".to_string() });\n ctx.db.user().insert(User { user_id: 2, name: \"Bob\".to_string() });\n\n // Insert groups\n ctx.db.group().insert(Group { group_id: 10, title: \"Admin\".to_string() });\n ctx.db.group().insert(Group { group_id: 20, title: \"Dev\".to_string() });\n\n // Insert memberships\n ctx.db.membership().insert(Membership { id: 1, user_id: 1, group_id: 10 });\n ctx.db.membership().insert(Membership { id: 2, user_id: 1, group_id: 20 });\n ctx.db.membership().insert(Membership { id: 3, user_id: 2, group_id: 20 });\n}", + "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = user)]\npub struct User {\n #[primary_key]\n user_id: i32,\n name: String,\n}\n\n#[spacetimedb::table(name = group)]\npub struct Group {\n #[primary_key]\n group_id: i32,\n title: String,\n}\n\n#[spacetimedb::table(\n name = membership,\n index(name = by_user, btree(columns = [user_id])),\n index(name = by_group, btree(columns = [group_id]))\n)]\npub struct Membership {\n #[primary_key]\n id: i32,\n user_id: i32,\n group_id: i32,\n}\n\n#[spacetimedb::reducer]\npub fn seed(ctx: &ReducerContext) {\n let membership = ctx.db.membership();\n for row in membership.iter() {\n membership.id().delete(&row.id);\n }\n\n let group = ctx.db.group();\n for row in group.iter() {\n group.group_id().delete(&row.group_id);\n }\n\n let user = ctx.db.user();\n for row in user.iter() {\n user.user_id().delete(&row.user_id);\n }\n\n user.insert(User { user_id: 1, name: \"Alice\".to_string() });\n user.insert(User { user_id: 2, name: \"Bob\".to_string() });\n\n group.insert(Group { group_id: 10, title: \"Admin\".to_string() });\n group.insert(Group { group_id: 20, title: \"Dev\".to_string() });\n\n membership.insert(Membership { id: 1, user_id: 1, group_id: 10 });\n membership.insert(Membership { id: 2, user_id: 1, group_id: 20 });\n membership.insert(Membership { id: 3, user_id: 2, group_id: 20 });\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-019-many-to-many-golden", @@ -950,13 +984,13 @@ "work_dir_golden": "target/llm-runs/schema/t_019_many_to_many/rust/server/golden", "work_dir_llm": "target/llm-runs/schema/t_019_many_to_many/rust/server/gpt-5/llm", "scorer_details": { - "m2m_has_2_20": { + "m2m_has_1_10": { "pass": true, "partial": 1.0, "notes": { "actual": 1, "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM membership WHERE user_id=2 AND group_id=20" + "sql": "SELECT COUNT(*) AS n FROM membership WHERE user_id=1 AND group_id=10" } }, "schema_parity": { @@ -967,7 +1001,7 @@ "llm_db": "schema-t-019-many-to-many-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:33615", + "server": "http://127.0.0.1:45511", "tables_diff": null, "tables_equal": true } @@ -981,28 +1015,28 @@ "sql": "SELECT COUNT(*) AS n FROM membership WHERE user_id=1 AND group_id=20" } }, - "m2m_has_1_10": { + "memberships_three_rows": { "pass": true, "partial": 1.0, "notes": { - "actual": 1, - "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM membership WHERE user_id=1 AND group_id=10" + "actual": 3, + "expected": 3, + "sql": "SELECT COUNT(*) AS n FROM membership" } }, - "memberships_three_rows": { + "m2m_has_2_20": { "pass": true, "partial": 1.0, "notes": { - "actual": 3, - "expected": 3, - "sql": "SELECT COUNT(*) AS n FROM membership" + "actual": 1, + "expected": 1, + "sql": "SELECT COUNT(*) AS n FROM membership WHERE user_id=2 AND group_id=20" } } }, "vendor": "openai", - "started_at": "2026-01-27T19:49:28.665682949Z", - "finished_at": "2026-01-27T19:50:22.268882588Z" + "started_at": "2026-01-27T21:56:26.171964152Z", + "finished_at": "2026-01-27T21:57:30.262139056Z" }, "t_020_ecs": { "hash": "2a0f4ec42cac03fb3da542793b311e4b399845c688bb0660a6c231c82e96c56a", @@ -1012,7 +1046,7 @@ "model_name": "GPT-5", "total_tests": 5, "passed_tests": 1, - "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table};\n\n#[table(name = entity)]\npub struct Entity {\n #[primary_key]\n id: i32,\n}\n\n#[table(name = position)]\npub struct Position {\n #[primary_key]\n entity_id: i32,\n x: i32,\n y: i32,\n}\n\n#[table(name = velocity)]\npub struct Velocity {\n #[primary_key]\n entity_id: i32,\n vx: i32,\n vy: i32,\n}\n\n#[table(name = next_position)]\npub struct NextPosition {\n #[primary_key]\n entity_id: i32,\n x: i32,\n y: i32,\n}\n\n#[reducer]\npub fn seed(ctx: &ReducerContext) {\n let _ = ctx.db.entity().try_insert(Entity { id: 1 });\n let _ = ctx.db.entity().try_insert(Entity { id: 2 });\n\n let _ = ctx.db.position().try_insert(Position { entity_id: 1, x: 0, y: 0 });\n let _ = ctx.db.velocity().try_insert(Velocity { entity_id: 1, vx: 1, vy: 0 });\n\n let _ = ctx.db.position().try_insert(Position { entity_id: 2, x: 10, y: 0 });\n let _ = ctx.db.velocity().try_insert(Velocity { entity_id: 2, vx: -2, vy: 3 });\n}\n\n#[reducer]\npub fn step(ctx: &ReducerContext) {\n for pos in ctx.db.position().iter() {\n if let Some(vel) = ctx.db.velocity().entity_id().find(pos.entity_id) {\n let nx = pos.x + vel.vx;\n let ny = pos.y + vel.vy;\n\n if let Some(mut existing) = ctx.db.next_position().entity_id().find(pos.entity_id) {\n existing.x = nx;\n existing.y = ny;\n let _ = ctx.db.next_position().entity_id().update(existing);\n } else {\n let _ = ctx.db.next_position().try_insert(NextPosition {\n entity_id: pos.entity_id,\n x: nx,\n y: ny,\n });\n }\n }\n }\n}", + "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table, UniqueColumn};\n\n#[table(name = entity)]\npub struct Entity {\n #[primary_key]\n id: i32,\n}\n\n#[table(name = position)]\npub struct Position {\n #[primary_key]\n entity_id: i32,\n x: i32,\n y: i32,\n}\n\n#[table(name = velocity)]\npub struct Velocity {\n #[primary_key]\n entity_id: i32,\n vx: i32,\n vy: i32,\n}\n\n#[table(name = next_position)]\npub struct NextPosition {\n #[primary_key]\n entity_id: i32,\n x: i32,\n y: i32,\n}\n\n#[reducer]\npub fn seed(ctx: &ReducerContext) {\n let _ = ctx.db.entity().try_insert(Entity { id: 1 });\n let _ = ctx.db.entity().try_insert(Entity { id: 2 });\n\n let _ = ctx.db.position().try_insert(Position { entity_id: 1, x: 0, y: 0 });\n let _ = ctx.db.velocity().try_insert(Velocity { entity_id: 1, vx: 1, vy: 0 });\n\n let _ = ctx.db.position().try_insert(Position { entity_id: 2, x: 10, y: 0 });\n let _ = ctx.db.velocity().try_insert(Velocity { entity_id: 2, vx: -2, vy: 3 });\n}\n\n#[reducer]\npub fn step(ctx: &ReducerContext) {\n let positions = ctx.db.position();\n let velocities = ctx.db.velocity();\n let next = ctx.db.next_position();\n\n for p in positions.iter() {\n if let Some(v) = velocities.entity_id().find(p.entity_id) {\n let nx = p.x + v.vx;\n let ny = p.y + v.vy;\n let row = NextPosition { entity_id: p.entity_id, x: nx, y: ny };\n if next.entity_id().find(p.entity_id).is_some() {\n let _ = next.entity_id().update(row);\n } else {\n let _ = next.insert(row);\n }\n }\n }\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-020-ecs-golden", @@ -1020,11 +1054,11 @@ "work_dir_golden": "target/llm-runs/schema/t_020_ecs/rust/server/golden", "work_dir_llm": "target/llm-runs/schema/t_020_ecs/rust/server/gpt-5/llm", "scorer_details": { - "ecs_step_next_positions_count": { + "ecs_seed_positions_count": { "pass": false, "partial": 0.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `next_positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:33615/v1/database/c200ef190fe9bac0d026751d082f663c75cb0435d300fdc35e1bc1f038529566/sql)\n", + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:45511/v1/database/c200f539c39322a26f09c5d34ffb0854ca4bb2f56b340ad9ddabef31e297f8de/sql)\n", "phase": "sql" } }, @@ -1032,43 +1066,43 @@ "pass": false, "partial": 0.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `next_positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:33615/v1/database/c200ef190fe9bac0d026751d082f663c75cb0435d300fdc35e1bc1f038529566/sql)\n", + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `next_positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:45511/v1/database/c200f539c39322a26f09c5d34ffb0854ca4bb2f56b340ad9ddabef31e297f8de/sql)\n", "phase": "sql" } }, - "schema_parity": { - "pass": true, - "partial": 1.0, + "ecs_step_next_positions_count": { + "pass": false, + "partial": 0.0, "notes": { - "golden_db": "schema-t-020-ecs-golden", - "llm_db": "schema-t-020-ecs-gpt-5-llm", - "reducers_diff": null, - "reducers_equal": true, - "server": "http://127.0.0.1:33615", - "tables_diff": null, - "tables_equal": true + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `next_positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:45511/v1/database/c200f539c39322a26f09c5d34ffb0854ca4bb2f56b340ad9ddabef31e297f8de/sql)\n", + "phase": "sql" } }, "ecs_next_pos_entity1": { "pass": false, "partial": 0.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `next_positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:33615/v1/database/c200ef190fe9bac0d026751d082f663c75cb0435d300fdc35e1bc1f038529566/sql)\n", + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `next_positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:45511/v1/database/c200f539c39322a26f09c5d34ffb0854ca4bb2f56b340ad9ddabef31e297f8de/sql)\n", "phase": "sql" } }, - "ecs_seed_positions_count": { - "pass": false, - "partial": 0.0, + "schema_parity": { + "pass": true, + "partial": 1.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:33615/v1/database/c200ef190fe9bac0d026751d082f663c75cb0435d300fdc35e1bc1f038529566/sql)\n", - "phase": "sql" + "golden_db": "schema-t-020-ecs-golden", + "llm_db": "schema-t-020-ecs-gpt-5-llm", + "reducers_diff": null, + "reducers_equal": true, + "server": "http://127.0.0.1:45511", + "tables_diff": null, + "tables_equal": true } } }, "vendor": "openai", - "started_at": "2026-01-27T19:49:28.665885964Z", - "finished_at": "2026-01-27T19:50:18.814108696Z" + "started_at": "2026-01-27T21:55:50.881999154Z", + "finished_at": "2026-01-27T21:56:52.270043237Z" }, "t_021_multi_column_index": { "hash": "2a0f4ec42cac03fb3da542793b311e4b399845c688bb0660a6c231c82e96c56a", @@ -1078,7 +1112,7 @@ "model_name": "GPT-5", "total_tests": 4, "passed_tests": 4, - "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table};\n\n#[table(name = log, index(name = by_user_day, btree(columns = [user_id, day])))]\npub struct Log {\n #[primary_key]\n id: i32,\n user_id: i32,\n day: i32,\n message: String,\n}\n\n#[reducer]\npub fn seed(ctx: &ReducerContext) {\n let table = ctx.db.log();\n table.insert(Log { id: 1, user_id: 7, day: 1, message: \"a\".to_string() });\n table.insert(Log { id: 2, user_id: 7, day: 2, message: \"b\".to_string() });\n table.insert(Log { id: 3, user_id: 9, day: 1, message: \"c\".to_string() });\n}", + "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = log, index(name = by_user_day, btree(columns = [user_id, day])))]\npub struct Log {\n #[primary_key]\n id: i32,\n user_id: i32,\n day: i32,\n message: String,\n}\n\n#[spacetimedb::reducer]\npub fn seed(ctx: &ReducerContext) {\n let table = ctx.db.log();\n let ids: Vec = table.iter().map(|row| row.id).collect();\n for id in ids {\n table.id().delete(&id);\n }\n table.insert(Log { id: 1, user_id: 7, day: 1, message: \"a\".to_string() });\n table.insert(Log { id: 2, user_id: 7, day: 2, message: \"b\".to_string() });\n table.insert(Log { id: 3, user_id: 9, day: 1, message: \"c\".to_string() });\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-021-multi-column-index-golden", @@ -1086,19 +1120,6 @@ "work_dir_golden": "target/llm-runs/schema/t_021_multi_column_index/rust/server/golden", "work_dir_llm": "target/llm-runs/schema/t_021_multi_column_index/rust/server/gpt-5/llm", "scorer_details": { - "schema_parity": { - "pass": true, - "partial": 1.0, - "notes": { - "golden_db": "schema-t-021-multi-column-index-golden", - "llm_db": "schema-t-021-multi-column-index-gpt-5-llm", - "reducers_diff": null, - "reducers_equal": true, - "server": "http://127.0.0.1:33615", - "tables_diff": null, - "tables_equal": true - } - }, "mcindex_seed_count": { "pass": true, "partial": 1.0, @@ -1117,7 +1138,20 @@ "sql": "SELECT COUNT(*) AS n FROM log WHERE user_id=7 AND day=1" } }, - "mcindex_lookup_u7_d2": { + "schema_parity": { + "pass": true, + "partial": 1.0, + "notes": { + "golden_db": "schema-t-021-multi-column-index-golden", + "llm_db": "schema-t-021-multi-column-index-gpt-5-llm", + "reducers_diff": null, + "reducers_equal": true, + "server": "http://127.0.0.1:45511", + "tables_diff": null, + "tables_equal": true + } + }, + "mcindex_lookup_u7_d2": { "pass": true, "partial": 1.0, "notes": { @@ -1128,8 +1162,8 @@ } }, "vendor": "openai", - "started_at": "2026-01-27T19:49:28.665048738Z", - "finished_at": "2026-01-27T19:50:12.818516478Z" + "started_at": "2026-01-27T21:55:50.881865222Z", + "finished_at": "2026-01-27T21:56:30.430284110Z" } } } @@ -1137,21 +1171,21 @@ }, { "mode": "docs", - "hash": "d1bf0b033db905c668d016082c762eb4b91edd1299f7b8d4bcc6def5398610f9", + "hash": "d86518d2ac099e60eb25705cd317e30df95f4d6a4787a2fe5aae12753b8ab4f7", "models": [ { "name": "GPT-5", "route_api_model": "gpt-5", "tasks": { "t_000_empty_reducers": { - "hash": "d1bf0b033db905c668d016082c762eb4b91edd1299f7b8d4bcc6def5398610f9", + "hash": "d86518d2ac099e60eb25705cd317e30df95f4d6a4787a2fe5aae12753b8ab4f7", "task": "t_000_empty_reducers", "lang": "rust", "golden_published": true, "model_name": "GPT-5", "total_tests": 1, "passed_tests": 0, - "llm_output": "use spacetimedb::ReducerContext;\n\n#[spacetimedb::reducer]\npub fn empty_reducer_no_args(ctx: &ReducerContext) {}\n\n#[spacetimedb::reducer]\npub fn empty_reducer_with_int(ctx: &ReducerContext, count: i32) {}\n\n#[spacetimedb::reducer]\npub fn empty_reducer_with_string(ctx: &ReducerContext, name: String) {}\n\n#[spacetimedb::reducer]\npub fn empty_reducer_with_two_args(ctx: &ReducerContext, count: i32, name: String) {}\n\n#[spacetimedb::reducer]\npub fn empty_reducer_with_three_args(ctx: &ReducerContext, active: bool, ratio: f32, label: String) {}", + "llm_output": "use spacetimedb::ReducerContext;\n\n#[spacetimedb::reducer]\npub fn empty_reducer_no_args(_ctx: &ReducerContext) { }\n\n#[spacetimedb::reducer]\npub fn empty_reducer_with_int(_ctx: &ReducerContext, count: i32) { }\n\n#[spacetimedb::reducer]\npub fn empty_reducer_with_string(_ctx: &ReducerContext, name: String) { }\n\n#[spacetimedb::reducer]\npub fn empty_reducer_with_two_args(_ctx: &ReducerContext, count: i32, name: String) { }\n\n#[spacetimedb::reducer]\npub fn empty_reducer_with_three_args(_ctx: &ReducerContext, active: bool, ratio: f32, label: String) { }", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-000-empty-reducers-golden", @@ -1169,18 +1203,18 @@ } }, "vendor": "openai", - "started_at": "2026-01-27T19:53:02.635478741Z", - "finished_at": "2026-01-27T19:53:40.787269737Z" + "started_at": "2026-01-27T21:59:28.738404956Z", + "finished_at": "2026-01-27T22:00:05.222156504Z" }, "t_001_basic_tables": { - "hash": "d1bf0b033db905c668d016082c762eb4b91edd1299f7b8d4bcc6def5398610f9", + "hash": "d86518d2ac099e60eb25705cd317e30df95f4d6a4787a2fe5aae12753b8ab4f7", "task": "t_001_basic_tables", "lang": "rust", "golden_published": true, "model_name": "GPT-5", "total_tests": 1, "passed_tests": 0, - "llm_output": "#[spacetimedb::table(name = user)]\npub struct User {\n #[primary_key]\n id: i32,\n name: String,\n age: i32,\n active: bool,\n}\n\n#[spacetimedb::table(name = product)]\npub struct Product {\n #[primary_key]\n id: i32,\n title: String,\n price: f32,\n in_stock: bool,\n}\n\n#[spacetimedb::table(name = note)]\npub struct Note {\n #[primary_key]\n id: i32,\n body: String,\n rating: i64,\n pinned: bool,\n}", + "llm_output": "use spacetimedb::Table;\n\n#[spacetimedb::table(name = user)]\npub struct User {\n #[primary_key]\n id: i32,\n name: String,\n age: i32,\n active: bool,\n}\n\n#[spacetimedb::table(name = product)]\npub struct Product {\n #[primary_key]\n id: i32,\n title: String,\n price: f32,\n in_stock: bool,\n}\n\n#[spacetimedb::table(name = note)]\npub struct Note {\n #[primary_key]\n id: i32,\n body: String,\n rating: i64,\n pinned: bool,\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-001-basic-tables-golden", @@ -1198,18 +1232,18 @@ } }, "vendor": "openai", - "started_at": "2026-01-27T19:53:04.338845678Z", - "finished_at": "2026-01-27T19:53:39.243226783Z" + "started_at": "2026-01-27T21:59:40.437506321Z", + "finished_at": "2026-01-27T22:00:09.706859982Z" }, "t_002_scheduled_table": { - "hash": "d1bf0b033db905c668d016082c762eb4b91edd1299f7b8d4bcc6def5398610f9", + "hash": "d86518d2ac099e60eb25705cd317e30df95f4d6a4787a2fe5aae12753b8ab4f7", "task": "t_002_scheduled_table", "lang": "rust", "golden_published": true, "model_name": "GPT-5", "total_tests": 1, "passed_tests": 0, - "llm_output": "use spacetimedb::{ReducerContext, ScheduleAt, Table};\n\n#[spacetimedb::table(name = tick_timer, scheduled(tick))]\npub struct TickTimer {\n #[primary_key]\n #[auto_inc]\n scheduled_id: u64,\n scheduled_at: ScheduleAt,\n}\n\n#[spacetimedb::reducer]\npub fn tick(_ctx: &ReducerContext, _timer: TickTimer) -> Result<(), String> {\n Ok(())\n}\n\n#[spacetimedb::reducer(init)]\npub fn init(ctx: &ReducerContext) -> Result<(), String> {\n if ctx.db.tick_timer().count() == 0 {\n use std::time::Duration;\n ctx.db.tick_timer().insert(TickTimer {\n scheduled_id: 0,\n scheduled_at: ScheduleAt::Interval(Duration::from_micros(50_000).into()),\n });\n }\n Ok(())\n}", + "llm_output": "use std::time::Duration;\nuse spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = tick_timer, scheduled(tick))]\npub struct TickTimer {\n #[primary_key]\n #[auto_inc]\n scheduled_id: u64,\n scheduled_at: spacetimedb::ScheduleAt,\n}\n\n#[spacetimedb::reducer(init)]\npub fn init(ctx: &ReducerContext) {\n if ctx.db.tick_timer().count() == 0 {\n ctx.db.tick_timer().insert(TickTimer {\n scheduled_id: 0,\n scheduled_at: spacetimedb::ScheduleAt::Interval(Duration::from_micros(50_000).into()),\n });\n }\n}\n\n#[spacetimedb::reducer]\npub fn tick(_ctx: &ReducerContext, _timer: TickTimer) {}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-002-scheduled-table-golden", @@ -1227,11 +1261,11 @@ } }, "vendor": "openai", - "started_at": "2026-01-27T19:53:38.403872273Z", - "finished_at": "2026-01-27T19:54:31.831288022Z" + "started_at": "2026-01-27T22:00:09.776019508Z", + "finished_at": "2026-01-27T22:01:03.100111948Z" }, "t_003_struct_in_table": { - "hash": "d1bf0b033db905c668d016082c762eb4b91edd1299f7b8d4bcc6def5398610f9", + "hash": "d86518d2ac099e60eb25705cd317e30df95f4d6a4787a2fe5aae12753b8ab4f7", "task": "t_003_struct_in_table", "lang": "rust", "golden_published": true, @@ -1256,11 +1290,11 @@ } }, "vendor": "openai", - "started_at": "2026-01-27T19:53:02.638916802Z", - "finished_at": "2026-01-27T19:53:40.055864699Z" + "started_at": "2026-01-27T22:00:09.779396910Z", + "finished_at": "2026-01-27T22:00:39.022195202Z" }, "t_004_insert": { - "hash": "d1bf0b033db905c668d016082c762eb4b91edd1299f7b8d4bcc6def5398610f9", + "hash": "d86518d2ac099e60eb25705cd317e30df95f4d6a4787a2fe5aae12753b8ab4f7", "task": "t_004_insert", "lang": "rust", "golden_published": true, @@ -1275,29 +1309,29 @@ "work_dir_golden": "target/llm-runs/basics/t_004_insert/rust/server/golden", "work_dir_llm": "target/llm-runs/basics/t_004_insert/rust/server/gpt-5/llm", "scorer_details": { - "schema_parity": { + "data_parity_insert_user": { "pass": false, "partial": 0.0, "notes": { - "error": "describe failed: WARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `basics-t-004-insert-golden`.\n", - "phase": "describe_golden" + "error": "spacetime call failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `basics-t-004-insert-golden`.\n", + "phase": "call_reducer_golden" } }, - "data_parity_insert_user": { + "schema_parity": { "pass": false, "partial": 0.0, "notes": { - "error": "spacetime call failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `basics-t-004-insert-golden`.\n", - "phase": "call_reducer_golden" + "error": "describe failed: WARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `basics-t-004-insert-golden`.\n", + "phase": "describe_golden" } } }, "vendor": "openai", - "started_at": "2026-01-27T19:53:33.748279444Z", - "finished_at": "2026-01-27T19:53:56.014195443Z" + "started_at": "2026-01-27T21:59:33.320363016Z", + "finished_at": "2026-01-27T22:00:09.225655385Z" }, "t_005_update": { - "hash": "d1bf0b033db905c668d016082c762eb4b91edd1299f7b8d4bcc6def5398610f9", + "hash": "d86518d2ac099e60eb25705cd317e30df95f4d6a4787a2fe5aae12753b8ab4f7", "task": "t_005_update", "lang": "rust", "golden_published": true, @@ -1339,11 +1373,11 @@ } }, "vendor": "openai", - "started_at": "2026-01-27T19:53:00.578961998Z", - "finished_at": "2026-01-27T19:53:38.403844461Z" + "started_at": "2026-01-27T21:59:28.647622669Z", + "finished_at": "2026-01-27T22:00:17.523842310Z" }, "t_006_delete": { - "hash": "d1bf0b033db905c668d016082c762eb4b91edd1299f7b8d4bcc6def5398610f9", + "hash": "d86518d2ac099e60eb25705cd317e30df95f4d6a4787a2fe5aae12753b8ab4f7", "task": "t_006_delete", "lang": "rust", "golden_published": true, @@ -1386,18 +1420,18 @@ } }, "vendor": "openai", - "started_at": "2026-01-27T19:52:31.910165978Z", - "finished_at": "2026-01-27T19:52:58.192517311Z" + "started_at": "2026-01-27T21:59:34.492198269Z", + "finished_at": "2026-01-27T22:00:08.795354268Z" }, "t_007_crud": { - "hash": "d1bf0b033db905c668d016082c762eb4b91edd1299f7b8d4bcc6def5398610f9", + "hash": "d86518d2ac099e60eb25705cd317e30df95f4d6a4787a2fe5aae12753b8ab4f7", "task": "t_007_crud", "lang": "rust", "golden_published": true, "model_name": "GPT-5", "total_tests": 4, "passed_tests": 1, - "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = user)]\npub struct User {\n #[primary_key]\n id: i32,\n name: String,\n age: i32,\n active: bool,\n}\n\n#[spacetimedb::reducer]\npub fn crud(ctx: &ReducerContext) {\n ctx.db.user().insert(User { id: 1, name: \"Alice\".into(), age: 30, active: true });\n ctx.db.user().insert(User { id: 2, name: \"Bob\".into(), age: 22, active: false });\n ctx.db.user().id().update(User { id: 1, name: \"Alice2\".into(), age: 31, active: false });\n ctx.db.user().id().delete(2);\n}", + "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = user)]\npub struct User {\n #[primary_key]\n id: i32,\n name: String,\n age: i32,\n active: bool,\n}\n\n#[spacetimedb::reducer]\npub fn crud(ctx: &ReducerContext) {\n ctx.db.user().insert(User { id: 1, name: \"Alice\".to_string(), age: 30, active: true });\n ctx.db.user().insert(User { id: 2, name: \"Bob\".to_string(), age: 22, active: false });\n\n if let Some(mut u) = ctx.db.user().id().find(1) {\n u.name = \"Alice2\".to_string();\n u.age = 31;\n u.active = false;\n ctx.db.user().id().update(u);\n }\n\n ctx.db.user().id().delete(2);\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-007-crud-golden", @@ -1422,14 +1456,6 @@ "sql": "SELECT COUNT(*) AS n FROM user" } }, - "schema_parity": { - "pass": false, - "partial": 0.0, - "notes": { - "error": "describe failed: WARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `basics-t-007-crud-golden`.\n", - "phase": "describe_golden" - } - }, "crud_row_id2_deleted": { "pass": true, "partial": 1.0, @@ -1438,21 +1464,37 @@ "expected": 0, "sql": "SELECT COUNT(*) AS n FROM user WHERE id=2" } + }, + "crud_row_id1_parity": { + "pass": false, + "partial": 0.0, + "notes": { + "error": "spacetime call failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `basics-t-007-crud-golden`.\n", + "phase": "call_reducer_golden" + } + }, + "schema_parity": { + "pass": false, + "partial": 0.0, + "notes": { + "error": "describe failed: WARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `basics-t-007-crud-golden`.\n", + "phase": "describe_golden" + } } }, "vendor": "openai", - "started_at": "2026-01-27T19:52:42.328837521Z", - "finished_at": "2026-01-27T19:53:33.740284067Z" + "started_at": "2026-01-27T22:00:09.225690701Z", + "finished_at": "2026-01-27T22:00:53.098859615Z" }, "t_008_index_lookup": { - "hash": "d1bf0b033db905c668d016082c762eb4b91edd1299f7b8d4bcc6def5398610f9", + "hash": "d86518d2ac099e60eb25705cd317e30df95f4d6a4787a2fe5aae12753b8ab4f7", "task": "t_008_index_lookup", "lang": "rust", "golden_published": true, "model_name": "GPT-5", "total_tests": 3, "passed_tests": 0, - "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = user)]\npub struct User {\n #[primary_key]\n id: i32,\n name: String,\n age: i32,\n active: bool,\n}\n\n#[spacetimedb::table(name = result)]\npub struct ResultRow {\n #[primary_key]\n id: i32,\n name: String,\n}\n\n#[spacetimedb::reducer]\npub fn lookup_user_name(ctx: &ReducerContext, id: i32) {\n if let Some(user) = ctx.db.user().id().find(id) {\n let row = ResultRow { id: user.id, name: user.name.clone() };\n if ctx.db.result().id().find(user.id).is_some() {\n ctx.db.result().id().update(row);\n } else {\n ctx.db.result().insert(row);\n }\n }\n}", + "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = user)]\npub struct User {\n #[primary_key]\n id: i32,\n name: String,\n age: i32,\n active: bool,\n}\n\n#[spacetimedb::table(name = result)]\npub struct ResultRow {\n #[primary_key]\n id: i32,\n name: String,\n}\n\n#[spacetimedb::reducer]\npub fn lookup_user_name(ctx: &ReducerContext, id: i32) {\n if let Some(user) = ctx.db.user().id().find(id) {\n if ctx.db.result().id().find(id).is_some() {\n ctx.db.result().id().update(ResultRow { id, name: user.name });\n } else {\n ctx.db.result().insert(ResultRow { id, name: user.name });\n }\n }\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-008-index-lookup-golden", @@ -1487,11 +1529,11 @@ } }, "vendor": "openai", - "started_at": "2026-01-27T19:53:17.657026192Z", - "finished_at": "2026-01-27T19:53:54.093778910Z" + "started_at": "2026-01-27T22:00:08.795543885Z", + "finished_at": "2026-01-27T22:00:42.342192764Z" }, "t_009_init": { - "hash": "d1bf0b033db905c668d016082c762eb4b91edd1299f7b8d4bcc6def5398610f9", + "hash": "d86518d2ac099e60eb25705cd317e30df95f4d6a4787a2fe5aae12753b8ab4f7", "task": "t_009_init", "lang": "rust", "golden_published": true, @@ -1506,18 +1548,9 @@ "work_dir_golden": "target/llm-runs/basics/t_009_init/rust/server/golden", "work_dir_llm": "target/llm-runs/basics/t_009_init/rust/server/gpt-5/llm", "scorer_details": { - "init_seed_bob": { + "init_seed_alice": { "pass": true, "partial": 1.0, - "notes": { - "actual": 1, - "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM user WHERE id=2 AND name='Bob' AND age=22 AND active=false" - } - }, - "schema_parity": { - "pass": false, - "partial": 0.0, "notes": { "error": "describe failed: WARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `basics-t-009-init-golden`.\n", "phase": "describe_golden" @@ -1532,9 +1565,18 @@ "sql": "SELECT COUNT(*) AS n FROM user" } }, - "init_seed_alice": { + "init_seed_bob": { "pass": true, "partial": 1.0, + "notes": { + "actual": 1, + "expected": 1, + "sql": "SELECT COUNT(*) AS n FROM user WHERE id=2 AND name='Bob' AND age=22 AND active=false" + } + }, + "schema_parity": { + "pass": false, + "partial": 0.0, "notes": { "actual": 1, "expected": 1, @@ -1543,18 +1585,18 @@ } }, "vendor": "openai", - "started_at": "2026-01-27T19:52:58.192552680Z", - "finished_at": "2026-01-27T19:53:38.342926441Z" + "started_at": "2026-01-27T21:59:50.287538471Z", + "finished_at": "2026-01-27T22:00:17.757690411Z" }, "t_010_connect": { - "hash": "d1bf0b033db905c668d016082c762eb4b91edd1299f7b8d4bcc6def5398610f9", + "hash": "d86518d2ac099e60eb25705cd317e30df95f4d6a4787a2fe5aae12753b8ab4f7", "task": "t_010_connect", "lang": "rust", "golden_published": true, "model_name": "GPT-5", "total_tests": 1, "passed_tests": 0, - "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = event)]\npub struct Event {\n #[primary_key]\n #[auto_inc]\n id: i32,\n kind: String,\n}\n\n#[spacetimedb::reducer(client_connected)]\npub fn client_connected(ctx: &ReducerContext) {\n ctx.db.event().insert(Event { id: 0, kind: \"connected\".to_string() });\n}\n\n#[spacetimedb::reducer(client_disconnected)]\npub fn client_disconnected(ctx: &ReducerContext) {\n ctx.db.event().insert(Event { id: 0, kind: \"disconnected\".to_string() });\n}", + "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = event)]\npub struct Event {\n #[primary_key]\n #[auto_inc]\n id: i32,\n kind: String,\n}\n\n#[spacetimedb::reducer(client_connected)]\npub fn client_connected(ctx: &ReducerContext) -> Result<(), String> {\n ctx.db.event().insert(Event { id: 0, kind: \"connected\".into() });\n Ok(())\n}\n\n#[spacetimedb::reducer(client_disconnected)]\npub fn client_disconnected(ctx: &ReducerContext) -> Result<(), String> {\n ctx.db.event().insert(Event { id: 0, kind: \"disconnected\".into() });\n Ok(())\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-010-connect-golden", @@ -1572,11 +1614,11 @@ } }, "vendor": "openai", - "started_at": "2026-01-27T19:52:35.522740907Z", - "finished_at": "2026-01-27T19:53:02.635440935Z" + "started_at": "2026-01-27T22:00:05.222185543Z", + "finished_at": "2026-01-27T22:00:38.284150630Z" }, "t_011_helper_function": { - "hash": "d1bf0b033db905c668d016082c762eb4b91edd1299f7b8d4bcc6def5398610f9", + "hash": "d86518d2ac099e60eb25705cd317e30df95f4d6a4787a2fe5aae12753b8ab4f7", "task": "t_011_helper_function", "lang": "rust", "golden_published": true, @@ -1618,18 +1660,18 @@ } }, "vendor": "openai", - "started_at": "2026-01-27T19:53:37.773674958Z", - "finished_at": "2026-01-27T19:53:57.689433966Z" + "started_at": "2026-01-27T21:59:47.134179939Z", + "finished_at": "2026-01-27T22:00:09.775866314Z" }, "t_012_spacetime_product_type": { - "hash": "d1bf0b033db905c668d016082c762eb4b91edd1299f7b8d4bcc6def5398610f9", + "hash": "d86518d2ac099e60eb25705cd317e30df95f4d6a4787a2fe5aae12753b8ab4f7", "task": "t_012_spacetime_product_type", "lang": "rust", "golden_published": true, "model_name": "GPT-5", "total_tests": 3, "passed_tests": 0, - "llm_output": "use spacetimedb::{ReducerContext, SpacetimeType, Table};\n\n#[derive(SpacetimeType)]\npub struct Score {\n left: i32,\n right: i32,\n}\n\n#[spacetimedb::table(name = result)]\npub struct ResultRow {\n #[primary_key]\n id: i32,\n value: Score,\n}\n\n#[spacetimedb::reducer]\npub fn set_score(ctx: &ReducerContext, id: i32, left: i32, right: i32) {\n ctx.db.result().insert(ResultRow {\n id,\n value: Score { left, right },\n });\n}", + "llm_output": "use spacetimedb::{ReducerContext, SpacetimeType, Table};\n\n#[derive(SpacetimeType, Clone, Debug)]\npub struct Score {\n left: i32,\n right: i32,\n}\n\n#[spacetimedb::table(name = result)]\npub struct ResultRow {\n #[primary_key]\n id: i32,\n value: Score,\n}\n\n#[spacetimedb::reducer]\npub fn set_score(ctx: &ReducerContext, id: i32, left: i32, right: i32) {\n ctx.db.result().insert(ResultRow {\n id,\n value: Score { left, right },\n });\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-012-spacetime-product-type-golden", @@ -1637,6 +1679,14 @@ "work_dir_golden": "target/llm-runs/schema/t_012_spacetime_product_type/rust/server/golden", "work_dir_llm": "target/llm-runs/schema/t_012_spacetime_product_type/rust/server/gpt-5/llm", "scorer_details": { + "schema_parity": { + "pass": false, + "partial": 0.0, + "notes": { + "error": "describe failed: WARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `schema-t-012-spacetime-product-type-golden`.\n", + "phase": "describe_golden" + } + }, "product_type_row_parity": { "pass": false, "partial": 0.0, @@ -1645,12 +1695,13 @@ "phase": "call_reducer_golden" } }, - "schema_parity": { + "product_type_row_count": { "pass": false, "partial": 0.0, "notes": { - "error": "describe failed: WARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `schema-t-012-spacetime-product-type-golden`.\n", - "phase": "describe_golden" + "actual": 0, + "expected": 1, + "sql": "SELECT COUNT(*) AS n FROM result WHERE id=1" } }, "product_type_row_count": { @@ -1664,18 +1715,18 @@ } }, "vendor": "openai", - "started_at": "2026-01-27T19:52:29.436388253Z", - "finished_at": "2026-01-27T19:53:04.338818518Z" + "started_at": "2026-01-27T21:58:34.113911342Z", + "finished_at": "2026-01-27T21:59:34.492169941Z" }, "t_013_spacetime_sum_type": { - "hash": "d1bf0b033db905c668d016082c762eb4b91edd1299f7b8d4bcc6def5398610f9", + "hash": "d86518d2ac099e60eb25705cd317e30df95f4d6a4787a2fe5aae12753b8ab4f7", "task": "t_013_spacetime_sum_type", "lang": "rust", "golden_published": true, "model_name": "GPT-5", "total_tests": 3, "passed_tests": 0, - "llm_output": "use spacetimedb::Table;\n\n#[derive(spacetimedb::SpacetimeType, Clone, Debug)]\npub struct Rect {\n width: i32,\n height: i32,\n}\n\n#[derive(spacetimedb::SpacetimeType, Clone, Debug)]\npub enum Shape {\n Circle(i32),\n Rectangle(Rect),\n}\n\n#[spacetimedb::table(name = result)]\npub struct ResultRow {\n #[primary_key]\n id: i32,\n value: Shape,\n}\n\n#[spacetimedb::reducer]\npub fn set_circle(ctx: &spacetimedb::ReducerContext, id: i32, radius: i32) {\n ctx.db.result().insert(ResultRow {\n id,\n value: Shape::Circle(radius),\n });\n}", + "llm_output": "use spacetimedb::{ReducerContext, SpacetimeType, Table};\n\n#[derive(SpacetimeType)]\npub struct Rect {\n width: i32,\n height: i32,\n}\n\n#[derive(SpacetimeType)]\npub enum Shape {\n Circle(i32),\n Rectangle(Rect),\n}\n\n#[spacetimedb::table(name = result)]\npub struct ResultRow {\n #[primary_key]\n id: i32,\n value: Shape,\n}\n\n#[spacetimedb::reducer]\npub fn set_circle(ctx: &ReducerContext, id: i32, radius: i32) {\n ctx.db.result().id().delete(id);\n ctx.db.result().insert(ResultRow {\n id,\n value: Shape::Circle(radius),\n });\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-013-spacetime-sum-type-golden", @@ -1691,37 +1742,37 @@ "phase": "describe_golden" } }, - "sum_type_row_count": { + "sum_type_row_parity": { "pass": false, "partial": 0.0, "notes": { - "actual": 0, - "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM result WHERE id=1" + "error": "describe failed: WARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `schema-t-013-spacetime-sum-type-golden`.\n", + "phase": "describe_golden" } }, - "sum_type_row_parity": { + "sum_type_row_count": { "pass": false, "partial": 0.0, "notes": { - "error": "spacetime call failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `schema-t-013-spacetime-sum-type-golden`.\n", - "phase": "call_reducer_golden" + "actual": 0, + "expected": 1, + "sql": "SELECT COUNT(*) AS n FROM result WHERE id=1" } } }, "vendor": "openai", - "started_at": "2026-01-27T19:51:47.638500741Z", - "finished_at": "2026-01-27T19:53:00.578888253Z" + "started_at": "2026-01-27T21:58:34.116466048Z", + "finished_at": "2026-01-27T21:59:07.957810697Z" }, "t_014_elementary_columns": { - "hash": "d1bf0b033db905c668d016082c762eb4b91edd1299f7b8d4bcc6def5398610f9", + "hash": "d86518d2ac099e60eb25705cd317e30df95f4d6a4787a2fe5aae12753b8ab4f7", "task": "t_014_elementary_columns", "lang": "rust", "golden_published": true, "model_name": "GPT-5", "total_tests": 3, "passed_tests": 0, - "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = primitive)]\npub struct Primitive {\n #[primary_key]\n id: i32,\n count: i32,\n total: i64,\n price: f32,\n ratio: f64,\n active: bool,\n name: String,\n}\n\n#[spacetimedb::reducer]\npub fn seed(ctx: &ReducerContext) {\n ctx.db.primitive().insert(Primitive {\n id: 1,\n count: 2,\n total: 3_000_000_000i64,\n price: 1.5f32,\n ratio: 2.25f64,\n active: true,\n name: \"Alice\".to_string(),\n });\n}", + "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = primitive)]\npub struct Primitive {\n #[primary_key]\n id: i32,\n count: i32,\n total: i64,\n price: f32,\n ratio: f64,\n active: bool,\n name: String,\n}\n\n#[spacetimedb::reducer]\npub fn seed(ctx: &ReducerContext) {\n ctx.db.primitive().insert(Primitive {\n id: 1,\n count: 2,\n total: 3000000000i64,\n price: 1.5f32,\n ratio: 2.25,\n active: true,\n name: \"Alice\".to_string(),\n });\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-014-elementary-columns-golden", @@ -1729,12 +1780,12 @@ "work_dir_golden": "target/llm-runs/schema/t_014_elementary_columns/rust/server/golden", "work_dir_llm": "target/llm-runs/schema/t_014_elementary_columns/rust/server/gpt-5/llm", "scorer_details": { - "elementary_columns_row_parity": { + "schema_parity": { "pass": false, "partial": 0.0, "notes": { - "error": "spacetime call failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `schema-t-014-elementary-columns-golden`.\n", - "phase": "call_reducer_golden" + "error": "describe failed: WARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `schema-t-014-elementary-columns-golden`.\n", + "phase": "describe_golden" } }, "elementary_columns_row_count": { @@ -1746,28 +1797,28 @@ "sql": "SELECT COUNT(*) AS n FROM primitive WHERE id=1" } }, - "schema_parity": { + "elementary_columns_row_parity": { "pass": false, "partial": 0.0, "notes": { - "error": "describe failed: WARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `schema-t-014-elementary-columns-golden`.\n", - "phase": "describe_golden" + "error": "spacetime call failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `schema-t-014-elementary-columns-golden`.\n", + "phase": "call_reducer_golden" } } }, "vendor": "openai", - "started_at": "2026-01-27T19:52:27.568323309Z", - "finished_at": "2026-01-27T19:53:02.605112864Z" + "started_at": "2026-01-27T21:59:07.957842631Z", + "finished_at": "2026-01-27T21:59:40.437477451Z" }, "t_015_product_type_columns": { - "hash": "d1bf0b033db905c668d016082c762eb4b91edd1299f7b8d4bcc6def5398610f9", + "hash": "d86518d2ac099e60eb25705cd317e30df95f4d6a4787a2fe5aae12753b8ab4f7", "task": "t_015_product_type_columns", "lang": "rust", "golden_published": false, "model_name": "GPT-5", "total_tests": 1, "passed_tests": 0, - "llm_output": null, + "llm_output": "use spacetimedb::{ReducerContext, SpacetimeType, Table};\n\n#[derive(SpacetimeType, Clone)]\npub struct Address {\n street: String,\n zip: i32,\n}\n\n#[derive(SpacetimeType, Clone)]\npub struct Position {\n x: i32,\n y: i32,\n}\n\n#[spacetimedb::table(name = profile)]\npub struct Profile {\n #[primary_key]\n id: i32,\n home: Address,\n work: Address,\n pos: Position,\n}\n\n#[spacetimedb::reducer]\npub fn seed(ctx: &ReducerContext) {\n if ctx.db.profile().id().find(1).is_none() {\n ctx.db.profile().insert(Profile {\n id: 1,\n home: Address {\n street: \"1 Main\".to_string(),\n zip: 11111,\n },\n work: Address {\n street: \"2 Broad\".to_string(),\n zip: 22222,\n },\n pos: Position { x: 7, y: 9 },\n });\n }\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": null, @@ -1775,28 +1826,45 @@ "work_dir_golden": null, "work_dir_llm": null, "scorer_details": { - "publish_error": { + "schema_parity": { "pass": false, "partial": 0.0, "notes": { - "error": "LLM call timed out", - "phase": "build_or_publish" + "error": "describe failed: WARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `schema-t-015-product-type-columns-golden`.\n", + "phase": "describe_golden" + } + }, + "product_type_columns_row_parity": { + "pass": false, + "partial": 0.0, + "notes": { + "error": "spacetime call failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `schema-t-015-product-type-columns-golden`.\n", + "phase": "call_reducer_golden" + } + }, + "product_type_columns_row_count": { + "pass": false, + "partial": 0.0, + "notes": { + "actual": 0, + "expected": 1, + "sql": "SELECT COUNT(*) AS n FROM profile WHERE id=1" } } }, "vendor": "openai", - "started_at": "2026-01-27T19:54:31.831326960Z", - "finished_at": "2026-01-27T19:54:31.831326960Z" + "started_at": "2026-01-27T21:58:34.132272186Z", + "finished_at": "2026-01-27T21:59:33.320275639Z" }, "t_016_sum_type_columns": { - "hash": "d1bf0b033db905c668d016082c762eb4b91edd1299f7b8d4bcc6def5398610f9", + "hash": "d86518d2ac099e60eb25705cd317e30df95f4d6a4787a2fe5aae12753b8ab4f7", "task": "t_016_sum_type_columns", "lang": "rust", "golden_published": true, "model_name": "GPT-5", "total_tests": 3, "passed_tests": 0, - "llm_output": "use spacetimedb::{ReducerContext, SpacetimeType, Table};\n\n#[derive(SpacetimeType, Clone, Debug)]\npub struct Rect {\n width: i32,\n height: i32,\n}\n\n#[derive(SpacetimeType, Clone, Debug)]\npub enum Shape {\n Circle(i32),\n Rectangle(Rect),\n}\n\n#[spacetimedb::table(name = drawing)]\npub struct Drawing {\n #[primary_key]\n id: i32,\n a: Shape,\n b: Shape,\n}\n\n#[spacetimedb::reducer]\npub fn seed(ctx: &ReducerContext) {\n if ctx.db.drawing().count() == 0 {\n ctx.db.drawing().insert(Drawing {\n id: 1,\n a: Shape::Circle(10),\n b: Shape::Rectangle(Rect { width: 4, height: 6 }),\n });\n }\n}", + "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[derive(spacetimedb::SpacetimeType, Clone)]\npub struct Rect {\n width: i32,\n height: i32,\n}\n\n#[derive(spacetimedb::SpacetimeType, Clone)]\npub enum Shape {\n Circle(i32),\n Rectangle(Rect),\n}\n\n#[spacetimedb::table(name = drawing)]\npub struct Drawing {\n #[primary_key]\n id: i32,\n a: Shape,\n b: Shape,\n}\n\n#[spacetimedb::reducer]\npub fn seed(ctx: &ReducerContext) {\n ctx.db.drawing().insert(Drawing {\n id: 1,\n a: Shape::Circle(10),\n b: Shape::Rectangle(Rect { width: 4, height: 6 }),\n });\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-016-sum-type-columns-golden", @@ -1804,44 +1872,44 @@ "work_dir_golden": "target/llm-runs/schema/t_016_sum_type_columns/rust/server/golden", "work_dir_llm": "target/llm-runs/schema/t_016_sum_type_columns/rust/server/gpt-5/llm", "scorer_details": { - "schema_parity": { + "sum_type_columns_row_parity": { "pass": false, "partial": 0.0, "notes": { - "error": "describe failed: WARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `schema-t-016-sum-type-columns-golden`.\n", - "phase": "describe_golden" + "error": "spacetime call failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `schema-t-016-sum-type-columns-golden`.\n", + "phase": "call_reducer_golden" } }, - "sum_type_columns_row_parity": { + "schema_parity": { "pass": false, "partial": 0.0, "notes": { - "error": "spacetime call failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `schema-t-016-sum-type-columns-golden`.\n", - "phase": "call_reducer_golden" + "error": "describe failed: WARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `schema-t-016-sum-type-columns-golden`.\n", + "phase": "describe_golden" } }, "sum_type_columns_row_count": { "pass": false, "partial": 0.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `drawings`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:38927/v1/database/c200596b7e7f2350abeb00119177ef3448e7673481a61aeaa7c22e4dd952cb52/sql)\n", + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `drawings`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:45981/v1/database/c200620d156430770305d35225398f5bdccd8e1edab109eb44e4284657fdb0d3/sql)\n", "phase": "sql" } } }, "vendor": "openai", - "started_at": "2026-01-27T19:51:47.634170474Z", - "finished_at": "2026-01-27T19:52:27.568235316Z" + "started_at": "2026-01-27T21:58:34.119405994Z", + "finished_at": "2026-01-27T21:59:28.738324004Z" }, "t_017_scheduled_columns": { - "hash": "d1bf0b033db905c668d016082c762eb4b91edd1299f7b8d4bcc6def5398610f9", + "hash": "d86518d2ac099e60eb25705cd317e30df95f4d6a4787a2fe5aae12753b8ab4f7", "task": "t_017_scheduled_columns", "lang": "rust", "golden_published": true, "model_name": "GPT-5", "total_tests": 2, "passed_tests": 1, - "llm_output": "use std::time::Duration;\nuse spacetimedb::{ReducerContext, ScheduleAt, Table};\n\n#[spacetimedb::table(name = tick_timer, scheduled(tick))]\npub struct TickTimer {\n #[primary_key]\n #[auto_inc]\n scheduled_id: u64,\n scheduled_at: ScheduleAt,\n}\n\n#[spacetimedb::reducer]\npub fn tick(_ctx: &ReducerContext, _tick_timer: TickTimer) -> Result<(), String> {\n Ok(())\n}\n\n#[spacetimedb::reducer(init)]\npub fn init(ctx: &ReducerContext) -> Result<(), String> {\n if ctx.db.tick_timer().count() == 0 {\n ctx.db.tick_timer().try_insert(TickTimer {\n scheduled_id: 0,\n scheduled_at: ScheduleAt::Interval(Duration::from_micros(50_000).into()),\n })?;\n }\n Ok(())\n}", + "llm_output": "use spacetimedb::{ReducerContext, ScheduleAt, Table};\nuse std::time::Duration;\n\n#[spacetimedb::table(name = tick_timer, scheduled(tick))]\npub struct TickTimer {\n #[primary_key]\n #[auto_inc]\n scheduled_id: u64,\n scheduled_at: ScheduleAt,\n}\n\n#[spacetimedb::reducer]\npub fn tick(_ctx: &ReducerContext, _timer: TickTimer) {\n}\n\n#[spacetimedb::reducer(init)]\npub fn init(ctx: &ReducerContext) -> Result<(), String> {\n ctx.db.tick_timer().try_insert(TickTimer {\n scheduled_id: 0,\n scheduled_at: ScheduleAt::Interval(Duration::from_micros(50_000).into()),\n })?;\n Ok(())\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-017-scheduled-columns-golden", @@ -1868,18 +1936,18 @@ } }, "vendor": "openai", - "started_at": "2026-01-27T19:51:47.625840249Z", - "finished_at": "2026-01-27T19:52:35.522695487Z" + "started_at": "2026-01-27T21:58:34.135515898Z", + "finished_at": "2026-01-27T21:59:47.134125164Z" }, "t_018_constraints": { - "hash": "d1bf0b033db905c668d016082c762eb4b91edd1299f7b8d4bcc6def5398610f9", + "hash": "d86518d2ac099e60eb25705cd317e30df95f4d6a4787a2fe5aae12753b8ab4f7", "task": "t_018_constraints", "lang": "rust", "golden_published": true, "model_name": "GPT-5", "total_tests": 3, "passed_tests": 0, - "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = account, index(name = by_name, btree(columns = [name])))]\npub struct Account {\n #[primary_key]\n id: i32,\n #[unique]\n email: String,\n name: String,\n}\n\n#[spacetimedb::reducer]\npub fn seed(ctx: &ReducerContext) {\n let ids: Vec = ctx.db.account().iter().map(|a| a.id).collect();\n for id in ids {\n ctx.db.account().id().delete(id);\n }\n\n ctx.db.account().insert(Account {\n id: 1,\n email: \"a@example.com\".into(),\n name: \"Alice\".into(),\n });\n ctx.db.account().insert(Account {\n id: 2,\n email: \"b@example.com\".into(),\n name: \"Bob\".into(),\n });\n}", + "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = account, index(name = by_name, btree(columns = [name])))]\npub struct Account {\n #[primary_key]\n id: i32,\n #[unique]\n email: String,\n name: String,\n}\n\n#[spacetimedb::reducer]\npub fn seed(ctx: &ReducerContext) {\n for row in ctx.db.account().iter() {\n ctx.db.account().id().delete(row.id);\n }\n\n ctx.db.account().insert(Account {\n id: 1,\n email: \"a@example.com\".to_string(),\n name: \"Alice\".to_string(),\n });\n\n ctx.db.account().insert(Account {\n id: 2,\n email: \"b@example.com\".to_string(),\n name: \"Bob\".to_string(),\n });\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-018-constraints-golden", @@ -1887,23 +1955,23 @@ "work_dir_golden": "target/llm-runs/schema/t_018_constraints/rust/server/golden", "work_dir_llm": "target/llm-runs/schema/t_018_constraints/rust/server/gpt-5/llm", "scorer_details": { - "constraints_row_parity_after_seed": { + "schema_parity": { "pass": false, "partial": 0.0, "notes": { - "error": "spacetime call failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `schema-t-018-constraints-golden`.\n", - "phase": "call_reducer_golden" + "error": "describe failed: WARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `schema-t-018-constraints-golden`.\n", + "phase": "describe_golden" } }, - "schema_parity": { + "constraints_seed_two_rows": { "pass": false, "partial": 0.0, "notes": { - "error": "describe failed: WARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `schema-t-018-constraints-golden`.\n", - "phase": "describe_golden" + "error": "spacetime call failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `schema-t-018-constraints-golden`.\n", + "phase": "call_reducer_golden" } }, - "constraints_seed_two_rows": { + "constraints_row_parity_after_seed": { "pass": false, "partial": 0.0, "notes": { @@ -1914,18 +1982,18 @@ } }, "vendor": "openai", - "started_at": "2026-01-27T19:51:47.622424031Z", - "finished_at": "2026-01-27T19:53:37.773579225Z" + "started_at": "2026-01-27T21:58:34.122551959Z", + "finished_at": "2026-01-27T21:59:28.647541973Z" }, "t_019_many_to_many": { - "hash": "d1bf0b033db905c668d016082c762eb4b91edd1299f7b8d4bcc6def5398610f9", + "hash": "d86518d2ac099e60eb25705cd317e30df95f4d6a4787a2fe5aae12753b8ab4f7", "task": "t_019_many_to_many", "lang": "rust", "golden_published": true, "model_name": "GPT-5", "total_tests": 5, "passed_tests": 4, - "llm_output": "use spacetimedb::Table;\n\n#[spacetimedb::table(name = user, public)]\npub struct User {\n #[primary_key]\n user_id: i32,\n name: String,\n}\n\n#[spacetimedb::table(name = group, public)]\npub struct Group {\n #[primary_key]\n group_id: i32,\n title: String,\n}\n\n#[spacetimedb::table(\n name = membership,\n public,\n index(name = by_user, btree(columns = [user_id])),\n index(name = by_group, btree(columns = [group_id]))\n)]\npub struct Membership {\n #[primary_key]\n id: i32,\n user_id: i32,\n group_id: i32,\n}\n\n#[spacetimedb::reducer]\npub fn seed(ctx: &spacetimedb::ReducerContext) {\n ctx.db.user().insert(User { user_id: 1, name: \"Alice\".into() });\n ctx.db.user().insert(User { user_id: 2, name: \"Bob\".into() });\n\n ctx.db.group().insert(Group { group_id: 10, title: \"Admin\".into() });\n ctx.db.group().insert(Group { group_id: 20, title: \"Dev\".into() });\n\n ctx.db.membership().insert(Membership { id: 1, user_id: 1, group_id: 10 });\n ctx.db.membership().insert(Membership { id: 2, user_id: 1, group_id: 20 });\n ctx.db.membership().insert(Membership { id: 3, user_id: 2, group_id: 20 });\n}", + "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = user, public)]\npub struct User {\n #[primary_key]\n user_id: i32,\n name: String,\n}\n\n#[spacetimedb::table(name = group, public)]\npub struct Group {\n #[primary_key]\n group_id: i32,\n title: String,\n}\n\n#[spacetimedb::table(\n name = membership,\n public,\n index(name = by_user, btree(columns = [user_id])),\n index(name = by_group, btree(columns = [group_id]))\n)]\npub struct Membership {\n #[primary_key]\n id: i32,\n user_id: i32,\n group_id: i32,\n}\n\n#[spacetimedb::reducer]\npub fn seed(ctx: &ReducerContext) {\n // Clear existing rows to ensure we end up with exactly the specified data set\n for m in ctx.db.membership().iter().collect::>() {\n ctx.db.membership().id().delete(&m.id);\n }\n for g in ctx.db.r#group().iter().collect::>() {\n ctx.db.r#group().group_id().delete(&g.group_id);\n }\n for u in ctx.db.user().iter().collect::>() {\n ctx.db.user().user_id().delete(&u.user_id);\n }\n\n // Insert users\n ctx.db.user().insert(User {\n user_id: 1,\n name: \"Alice\".into(),\n });\n ctx.db.user().insert(User {\n user_id: 2,\n name: \"Bob\".into(),\n });\n\n // Insert groups\n ctx.db.r#group().insert(Group {\n group_id: 10,\n title: \"Admin\".into(),\n });\n ctx.db.r#group().insert(Group {\n group_id: 20,\n title: \"Dev\".into(),\n });\n\n // Insert memberships\n ctx.db.membership().insert(Membership {\n id: 1,\n user_id: 1,\n group_id: 10,\n });\n ctx.db.membership().insert(Membership {\n id: 2,\n user_id: 1,\n group_id: 20,\n });\n ctx.db.membership().insert(Membership {\n id: 3,\n user_id: 2,\n group_id: 20,\n });\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-019-many-to-many-golden", @@ -1933,23 +2001,6 @@ "work_dir_golden": "target/llm-runs/schema/t_019_many_to_many/rust/server/golden", "work_dir_llm": "target/llm-runs/schema/t_019_many_to_many/rust/server/gpt-5/llm", "scorer_details": { - "memberships_three_rows": { - "pass": true, - "partial": 1.0, - "notes": { - "actual": 3, - "expected": 3, - "sql": "SELECT COUNT(*) AS n FROM membership" - } - }, - "schema_parity": { - "pass": false, - "partial": 0.0, - "notes": { - "error": "describe failed: WARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `schema-t-019-many-to-many-golden`.\n", - "phase": "describe_golden" - } - }, "m2m_has_1_20": { "pass": true, "partial": 1.0, @@ -1976,21 +2027,38 @@ "expected": 1, "sql": "SELECT COUNT(*) AS n FROM membership WHERE user_id=2 AND group_id=20" } + }, + "memberships_three_rows": { + "pass": true, + "partial": 1.0, + "notes": { + "actual": 3, + "expected": 3, + "sql": "SELECT COUNT(*) AS n FROM membership" + } + }, + "schema_parity": { + "pass": false, + "partial": 0.0, + "notes": { + "error": "describe failed: WARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `schema-t-019-many-to-many-golden`.\n", + "phase": "describe_golden" + } } }, "vendor": "openai", - "started_at": "2026-01-27T19:51:47.642788029Z", - "finished_at": "2026-01-27T19:52:31.910080797Z" + "started_at": "2026-01-27T21:59:22.401446796Z", + "finished_at": "2026-01-27T22:00:29.002067239Z" }, "t_020_ecs": { - "hash": "d1bf0b033db905c668d016082c762eb4b91edd1299f7b8d4bcc6def5398610f9", + "hash": "d86518d2ac099e60eb25705cd317e30df95f4d6a4787a2fe5aae12753b8ab4f7", "task": "t_020_ecs", "lang": "rust", "golden_published": true, "model_name": "GPT-5", "total_tests": 5, "passed_tests": 0, - "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = entity)]\npub struct Entity {\n #[primary_key]\n id: i32,\n}\n\n#[spacetimedb::table(name = position)]\npub struct Position {\n #[primary_key]\n entity_id: i32,\n x: i32,\n y: i32,\n}\n\n#[spacetimedb::table(name = velocity)]\npub struct Velocity {\n #[primary_key]\n entity_id: i32,\n vx: i32,\n vy: i32,\n}\n\n#[spacetimedb::table(name = next_position)]\npub struct NextPosition {\n #[primary_key]\n entity_id: i32,\n x: i32,\n y: i32,\n}\n\n#[spacetimedb::reducer]\npub fn seed(ctx: &ReducerContext) {\n // First entity: id=1, pos=(0,0), vel=(1,0)\n ctx.db.entity().id().delete(1);\n ctx.db.position().entity_id().delete(1);\n ctx.db.velocity().entity_id().delete(1);\n ctx.db.entity().insert(Entity { id: 1 });\n ctx.db.position().insert(Position { entity_id: 1, x: 0, y: 0 });\n ctx.db.velocity().insert(Velocity { entity_id: 1, vx: 1, vy: 0 });\n\n // Second entity: id=2, pos=(10,0), vel=(-2,3)\n ctx.db.entity().id().delete(2);\n ctx.db.position().entity_id().delete(2);\n ctx.db.velocity().entity_id().delete(2);\n ctx.db.entity().insert(Entity { id: 2 });\n ctx.db.position().insert(Position { entity_id: 2, x: 10, y: 0 });\n ctx.db.velocity().insert(Velocity { entity_id: 2, vx: -2, vy: 3 });\n}\n\n#[spacetimedb::reducer]\npub fn step(ctx: &ReducerContext) {\n for pos in ctx.db.position().iter() {\n if let Some(vel) = ctx.db.velocity().entity_id().find(pos.entity_id) {\n let nx = pos.x + vel.vx;\n let ny = pos.y + vel.vy;\n if let Some(_existing) = ctx.db.next_position().entity_id().find(pos.entity_id) {\n ctx.db\n .next_position()\n .entity_id()\n .update(NextPosition { entity_id: pos.entity_id, x: nx, y: ny });\n } else {\n ctx.db\n .next_position()\n .insert(NextPosition { entity_id: pos.entity_id, x: nx, y: ny });\n }\n }\n }\n}", + "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = entity)]\npub struct Entity {\n #[primary_key]\n id: i32,\n}\n\n#[spacetimedb::table(name = position)]\npub struct Position {\n #[primary_key]\n entity_id: i32,\n x: i32,\n y: i32,\n}\n\n#[spacetimedb::table(name = velocity)]\npub struct Velocity {\n #[primary_key]\n entity_id: i32,\n vx: i32,\n vy: i32,\n}\n\n#[spacetimedb::table(name = next_position)]\npub struct NextPosition {\n #[primary_key]\n entity_id: i32,\n x: i32,\n y: i32,\n}\n\n#[spacetimedb::reducer]\npub fn seed(ctx: &ReducerContext) {\n // entity 1\n let _ = ctx.db.entity().try_insert(Entity { id: 1 });\n if let Some(mut p) = ctx.db.position().entity_id().find(1) {\n p.x = 0;\n p.y = 0;\n ctx.db.position().entity_id().update(p);\n } else {\n ctx.db.position().insert(Position { entity_id: 1, x: 0, y: 0 });\n }\n if let Some(mut v) = ctx.db.velocity().entity_id().find(1) {\n v.vx = 1;\n v.vy = 0;\n ctx.db.velocity().entity_id().update(v);\n } else {\n ctx.db.velocity().insert(Velocity { entity_id: 1, vx: 1, vy: 0 });\n }\n\n // entity 2\n let _ = ctx.db.entity().try_insert(Entity { id: 2 });\n if let Some(mut p) = ctx.db.position().entity_id().find(2) {\n p.x = 10;\n p.y = 0;\n ctx.db.position().entity_id().update(p);\n } else {\n ctx.db.position().insert(Position { entity_id: 2, x: 10, y: 0 });\n }\n if let Some(mut v) = ctx.db.velocity().entity_id().find(2) {\n v.vx = -2;\n v.vy = 3;\n ctx.db.velocity().entity_id().update(v);\n } else {\n ctx.db.velocity().insert(Velocity { entity_id: 2, vx: -2, vy: 3 });\n }\n}\n\n#[spacetimedb::reducer]\npub fn step(ctx: &ReducerContext) {\n for pos in ctx.db.position().iter() {\n if let Some(vel) = ctx.db.velocity().entity_id().find(pos.entity_id) {\n let nx = pos.x + vel.vx;\n let ny = pos.y + vel.vy;\n if let Some(mut np) = ctx.db.next_position().entity_id().find(pos.entity_id) {\n np.x = nx;\n np.y = ny;\n ctx.db.next_position().entity_id().update(np);\n } else {\n ctx.db.next_position().insert(NextPosition {\n entity_id: pos.entity_id,\n x: nx,\n y: ny,\n });\n }\n }\n }\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-020-ecs-golden", @@ -2010,23 +2078,23 @@ "pass": false, "partial": 0.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `next_positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:38927/v1/database/c2000dfb254023618a83b1182de85ff2f9590dae5ce1c9301d7d8cc01c8bff6e/sql)\n", + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `next_positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:45981/v1/database/c200af561dfe31790d5f75e7e6eb859acf922a8ce783c7752ecf0686e5aa8a05/sql)\n", "phase": "sql" } }, - "ecs_next_pos_entity1": { + "ecs_seed_positions_count": { "pass": false, "partial": 0.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `next_positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:38927/v1/database/c2000dfb254023618a83b1182de85ff2f9590dae5ce1c9301d7d8cc01c8bff6e/sql)\n", + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:45981/v1/database/c200af561dfe31790d5f75e7e6eb859acf922a8ce783c7752ecf0686e5aa8a05/sql)\n", "phase": "sql" } }, - "ecs_seed_positions_count": { + "ecs_next_pos_entity1": { "pass": false, "partial": 0.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:38927/v1/database/c2000dfb254023618a83b1182de85ff2f9590dae5ce1c9301d7d8cc01c8bff6e/sql)\n", + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `next_positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:45981/v1/database/c200af561dfe31790d5f75e7e6eb859acf922a8ce783c7752ecf0686e5aa8a05/sql)\n", "phase": "sql" } }, @@ -2034,24 +2102,24 @@ "pass": false, "partial": 0.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `next_positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:38927/v1/database/c2000dfb254023618a83b1182de85ff2f9590dae5ce1c9301d7d8cc01c8bff6e/sql)\n", + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `next_positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:45981/v1/database/c200af561dfe31790d5f75e7e6eb859acf922a8ce783c7752ecf0686e5aa8a05/sql)\n", "phase": "sql" } } }, "vendor": "openai", - "started_at": "2026-01-27T19:51:47.647113172Z", - "finished_at": "2026-01-27T19:52:42.328754582Z" + "started_at": "2026-01-27T21:58:34.128964652Z", + "finished_at": "2026-01-27T21:59:50.287487076Z" }, "t_021_multi_column_index": { - "hash": "d1bf0b033db905c668d016082c762eb4b91edd1299f7b8d4bcc6def5398610f9", + "hash": "d86518d2ac099e60eb25705cd317e30df95f4d6a4787a2fe5aae12753b8ab4f7", "task": "t_021_multi_column_index", "lang": "rust", "golden_published": true, "model_name": "GPT-5", "total_tests": 4, "passed_tests": 3, - "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = log, index(name = by_user_day, btree(columns = [user_id, day])))]\npub struct Log {\n #[primary_key]\n id: i32,\n user_id: i32,\n day: i32,\n message: String,\n}\n\n#[spacetimedb::reducer]\npub fn seed(ctx: &ReducerContext) {\n ctx.db.log().insert(Log { id: 1, user_id: 7, day: 1, message: \"a\".into() });\n ctx.db.log().insert(Log { id: 2, user_id: 7, day: 2, message: \"b\".into() });\n ctx.db.log().insert(Log { id: 3, user_id: 9, day: 1, message: \"c\".into() });\n}", + "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = log, index(name = by_user_day, btree(columns = [user_id, day])))]\npub struct Log {\n #[primary_key]\n id: i32,\n user_id: i32,\n day: i32,\n message: String,\n}\n\n#[spacetimedb::reducer]\npub fn seed(ctx: &ReducerContext) {\n let ids: Vec = ctx.db.log().iter().map(|r| r.id).collect();\n for id in ids {\n ctx.db.log().id().delete(&id);\n }\n ctx.db.log().insert(Log { id: 1, user_id: 7, day: 1, message: \"a\".to_string() });\n ctx.db.log().insert(Log { id: 2, user_id: 7, day: 2, message: \"b\".to_string() });\n ctx.db.log().insert(Log { id: 3, user_id: 9, day: 1, message: \"c\".to_string() });\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-021-multi-column-index-golden", @@ -2096,8 +2164,8 @@ } }, "vendor": "openai", - "started_at": "2026-01-27T19:51:47.629929093Z", - "finished_at": "2026-01-27T19:52:29.436282461Z" + "started_at": "2026-01-27T21:58:34.125789398Z", + "finished_at": "2026-01-27T21:59:22.401369156Z" } } } @@ -2288,14 +2356,14 @@ "modes": [ { "mode": "docs", - "hash": "e3621eaacfd5238fb514ff8c09c277ccd66170773959e5b391507110bf3ac17b", + "hash": "bd31092a0af7d25ac3ccd5ede135c9840ed8cad3c2d6006ee1ad0f5f69e3a3e1", "models": [ { "name": "GPT-5", "route_api_model": "gpt-5", "tasks": { "t_000_empty_reducers": { - "hash": "e3621eaacfd5238fb514ff8c09c277ccd66170773959e5b391507110bf3ac17b", + "hash": "bd31092a0af7d25ac3ccd5ede135c9840ed8cad3c2d6006ee1ad0f5f69e3a3e1", "task": "t_000_empty_reducers", "lang": "csharp", "golden_published": true, @@ -2318,25 +2386,25 @@ "llm_db": "basics-t-000-empty-reducers-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:34379", + "server": "http://127.0.0.1:42031", "tables_diff": null, "tables_equal": true } } }, "vendor": "openai", - "started_at": "2026-01-27T19:59:45.341740044Z", - "finished_at": "2026-01-27T20:00:17.110354447Z" + "started_at": "2026-01-27T22:05:00.112638697Z", + "finished_at": "2026-01-27T22:05:46.121861290Z" }, "t_001_basic_tables": { - "hash": "e3621eaacfd5238fb514ff8c09c277ccd66170773959e5b391507110bf3ac17b", + "hash": "bd31092a0af7d25ac3ccd5ede135c9840ed8cad3c2d6006ee1ad0f5f69e3a3e1", "task": "t_001_basic_tables", "lang": "csharp", "golden_published": true, "model_name": "GPT-5", "total_tests": 1, "passed_tests": 1, - "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"User\")]\n public partial struct User\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public string Name;\n public int Age;\n public bool Active;\n }\n\n [SpacetimeDB.Table(Name = \"Product\")]\n public partial struct Product\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public string Title;\n public float Price;\n public bool InStock;\n }\n\n [SpacetimeDB.Table(Name = \"Note\")]\n public partial struct Note\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public string Body;\n public long Rating;\n public bool Pinned;\n }\n}", + "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"User\", Public = true)]\n public partial struct User\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public string Name;\n public int Age;\n public bool Active;\n }\n\n [SpacetimeDB.Table(Name = \"Product\", Public = true)]\n public partial struct Product\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public string Title;\n public float Price;\n public bool InStock;\n }\n\n [SpacetimeDB.Table(Name = \"Note\", Public = true)]\n public partial struct Note\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public string Body;\n public long Rating;\n public bool Pinned;\n }\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-001-basic-tables-golden", @@ -2352,25 +2420,25 @@ "llm_db": "basics-t-001-basic-tables-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:34379", + "server": "http://127.0.0.1:42031", "tables_diff": null, "tables_equal": true } } }, "vendor": "openai", - "started_at": "2026-01-27T20:00:10.594319009Z", - "finished_at": "2026-01-27T20:00:39.454473930Z" + "started_at": "2026-01-27T22:05:46.121885196Z", + "finished_at": "2026-01-27T22:06:22.443409459Z" }, "t_002_scheduled_table": { - "hash": "e3621eaacfd5238fb514ff8c09c277ccd66170773959e5b391507110bf3ac17b", + "hash": "bd31092a0af7d25ac3ccd5ede135c9840ed8cad3c2d6006ee1ad0f5f69e3a3e1", "task": "t_002_scheduled_table", "lang": "csharp", "golden_published": true, "model_name": "GPT-5", "total_tests": 1, "passed_tests": 1, - "llm_output": "using System;\nusing SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"TickTimer\", Scheduled = nameof(Tick), ScheduledAt = nameof(ScheduledAt))]\n public partial struct TickTimer\n {\n [SpacetimeDB.PrimaryKey]\n [SpacetimeDB.AutoInc]\n public ulong ScheduledId;\n public ScheduleAt ScheduledAt;\n }\n\n [SpacetimeDB.Reducer(ReducerKind.Init)]\n public static void Init(ReducerContext ctx)\n {\n foreach (var row in ctx.Db.TickTimer.Iter())\n {\n ctx.Db.TickTimer.ScheduledId.Delete(row.ScheduledId);\n }\n\n ctx.Db.TickTimer.Insert(new TickTimer\n {\n ScheduledId = 0,\n ScheduledAt = new ScheduleAt.Interval(TimeSpan.FromMilliseconds(50))\n });\n }\n\n [SpacetimeDB.Reducer]\n public static void Tick(ReducerContext ctx, TickTimer _timer)\n {\n // Tick scheduled reducer. Intentionally left blank.\n }\n}", + "llm_output": "using System;\nusing SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"TickTimer\", Scheduled = nameof(Tick), ScheduledAt = nameof(ScheduledAt))]\n public partial struct TickTimer\n {\n [SpacetimeDB.PrimaryKey, SpacetimeDB.AutoInc]\n public ulong ScheduledId;\n public ScheduleAt ScheduledAt;\n }\n\n [SpacetimeDB.Reducer]\n public static void Tick(ReducerContext ctx, TickTimer timer)\n {\n }\n\n [SpacetimeDB.Reducer(ReducerKind.Init)]\n public static void Init(ReducerContext ctx)\n {\n if (ctx.Db.TickTimer.Count == 0)\n {\n ctx.Db.TickTimer.Insert(new TickTimer\n {\n ScheduledAt = new ScheduleAt.Interval(TimeSpan.FromMicroseconds(50_000))\n });\n }\n }\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-002-scheduled-table-golden", @@ -2386,18 +2454,18 @@ "llm_db": "basics-t-002-scheduled-table-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:34379", + "server": "http://127.0.0.1:42031", "tables_diff": null, "tables_equal": true } } }, "vendor": "openai", - "started_at": "2026-01-27T20:00:39.454500245Z", - "finished_at": "2026-01-27T20:01:19.695962602Z" + "started_at": "2026-01-27T22:06:22.566930178Z", + "finished_at": "2026-01-27T22:07:08.516744730Z" }, "t_003_struct_in_table": { - "hash": "e3621eaacfd5238fb514ff8c09c277ccd66170773959e5b391507110bf3ac17b", + "hash": "bd31092a0af7d25ac3ccd5ede135c9840ed8cad3c2d6006ee1ad0f5f69e3a3e1", "task": "t_003_struct_in_table", "lang": "csharp", "golden_published": true, @@ -2420,18 +2488,18 @@ "llm_db": "basics-t-003-struct-in-table-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:34379", + "server": "http://127.0.0.1:42031", "tables_diff": null, "tables_equal": true } } }, "vendor": "openai", - "started_at": "2026-01-27T20:00:06.267861248Z", - "finished_at": "2026-01-27T20:00:37.823969244Z" + "started_at": "2026-01-27T22:06:56.852214977Z", + "finished_at": "2026-01-27T22:07:18.733479497Z" }, "t_004_insert": { - "hash": "e3621eaacfd5238fb514ff8c09c277ccd66170773959e5b391507110bf3ac17b", + "hash": "bd31092a0af7d25ac3ccd5ede135c9840ed8cad3c2d6006ee1ad0f5f69e3a3e1", "task": "t_004_insert", "lang": "csharp", "golden_published": true, @@ -2454,7 +2522,7 @@ "llm_db": "basics-t-004-insert-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:34379", + "server": "http://127.0.0.1:42031", "tables_diff": null, "tables_equal": true } @@ -2475,23 +2543,23 @@ "llm_out": "Id | Name | Age | Active ----+---------+-----+-------- 1 | \"Alice\" | 30 | true", "query": "SELECT Id, Name, Age, Active FROM User WHERE Id=1", "reducer": "InsertUser", - "server": "http://127.0.0.1:34379" + "server": "http://127.0.0.1:42031" } } }, "vendor": "openai", - "started_at": "2026-01-27T20:00:17.336276015Z", - "finished_at": "2026-01-27T20:00:48.236577365Z" + "started_at": "2026-01-27T22:05:21.015736490Z", + "finished_at": "2026-01-27T22:05:47.405778904Z" }, "t_005_update": { - "hash": "e3621eaacfd5238fb514ff8c09c277ccd66170773959e5b391507110bf3ac17b", + "hash": "bd31092a0af7d25ac3ccd5ede135c9840ed8cad3c2d6006ee1ad0f5f69e3a3e1", "task": "t_005_update", "lang": "csharp", "golden_published": true, "model_name": "GPT-5", "total_tests": 3, "passed_tests": 3, - "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"User\")]\n public partial struct User\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public string Name;\n public int Age;\n public bool Active;\n }\n\n [SpacetimeDB.Reducer]\n public static void UpdateUser(ReducerContext ctx, int id, string name, int age, bool active)\n {\n var user = ctx.Db.User.Id.Find(id) ?? throw new System.Exception(\"User not found\");\n user.Name = name;\n user.Age = age;\n user.Active = active;\n user.Id = id;\n ctx.Db.User.Id.Update(user);\n }\n}", + "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"User\")]\n public partial struct User\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public string Name;\n public int Age;\n public bool Active;\n }\n\n [SpacetimeDB.Reducer]\n public static void UpdateUser(ReducerContext ctx, int id, string name, int age, bool active)\n {\n var row = ctx.Db.User.Id.Find(id);\n if (row is User)\n {\n ctx.Db.User.Id.Update(new User\n {\n Id = id,\n Name = name,\n Age = age,\n Active = active\n });\n }\n }\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-005-update-golden", @@ -2515,7 +2583,7 @@ "llm_out": "Id | Name | Age | Active ----+----------+-----+-------- 1 | \"Alice2\" | 31 | false", "query": "SELECT Id, Name, Age, Active FROM User WHERE Id=1", "reducer": "UpdateUser", - "server": "http://127.0.0.1:34379" + "server": "http://127.0.0.1:42031" } }, "seed_users_row": { @@ -2533,25 +2601,25 @@ "llm_db": "basics-t-005-update-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:34379", + "server": "http://127.0.0.1:42031", "tables_diff": null, "tables_equal": true } } }, "vendor": "openai", - "started_at": "2026-01-27T19:59:43.681559749Z", - "finished_at": "2026-01-27T20:00:17.332627669Z" + "started_at": "2026-01-27T22:04:50.875700846Z", + "finished_at": "2026-01-27T22:05:50.454024469Z" }, "t_006_delete": { - "hash": "e3621eaacfd5238fb514ff8c09c277ccd66170773959e5b391507110bf3ac17b", + "hash": "bd31092a0af7d25ac3ccd5ede135c9840ed8cad3c2d6006ee1ad0f5f69e3a3e1", "task": "t_006_delete", "lang": "csharp", "golden_published": true, "model_name": "GPT-5", "total_tests": 3, "passed_tests": 3, - "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"User\")]\n public partial struct User\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public string Name;\n public int Age;\n public bool Active;\n }\n\n [SpacetimeDB.Reducer]\n public static void DeleteUser(ReducerContext ctx, int id)\n {\n ctx.Db.User.Id.Delete(id);\n }\n}", + "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"User\", Public = true)]\n public partial struct User\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public string Name;\n public int Age;\n public bool Active;\n }\n\n [SpacetimeDB.Reducer]\n public static void DeleteUser(ReducerContext ctx, int id)\n {\n ctx.Db.User.Id.Delete(id);\n }\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-006-delete-golden", @@ -2559,11 +2627,17 @@ "work_dir_golden": "target/llm-runs/basics/t_006_delete/csharp/server/golden", "work_dir_llm": "target/llm-runs/basics/t_006_delete/csharp/server/gpt-5/llm", "scorer_details": { - "seed_users_row": { + "schema_parity": { "pass": true, "partial": 1.0, "notes": { - "sql": "INSERT INTO User(Id, Name, Age, Active) VALUES (1, 'Alice', 30, true)" + "golden_db": "basics-t-006-delete-golden", + "llm_db": "basics-t-006-delete-gpt-5-llm", + "reducers_diff": null, + "reducers_equal": true, + "server": "http://127.0.0.1:42031", + "tables_diff": null, + "tables_equal": true } }, "delete_user_count_zero": { @@ -2575,33 +2649,27 @@ "sql": "SELECT COUNT(*) AS n FROM User WHERE Id=1" } }, - "schema_parity": { + "seed_users_row": { "pass": true, "partial": 1.0, "notes": { - "golden_db": "basics-t-006-delete-golden", - "llm_db": "basics-t-006-delete-gpt-5-llm", - "reducers_diff": null, - "reducers_equal": true, - "server": "http://127.0.0.1:34379", - "tables_diff": null, - "tables_equal": true + "sql": "INSERT INTO User(Id, Name, Age, Active) VALUES (1, 'Alice', 30, true)" } } }, "vendor": "openai", - "started_at": "2026-01-27T19:58:58.264033858Z", - "finished_at": "2026-01-27T19:59:39.007925686Z" + "started_at": "2026-01-27T22:05:42.457690222Z", + "finished_at": "2026-01-27T22:06:15.967919960Z" }, "t_007_crud": { - "hash": "e3621eaacfd5238fb514ff8c09c277ccd66170773959e5b391507110bf3ac17b", + "hash": "bd31092a0af7d25ac3ccd5ede135c9840ed8cad3c2d6006ee1ad0f5f69e3a3e1", "task": "t_007_crud", "lang": "csharp", "golden_published": true, "model_name": "GPT-5", "total_tests": 4, "passed_tests": 4, - "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"User\")]\n public partial struct User\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public string Name;\n public int Age;\n public bool Active;\n }\n\n [SpacetimeDB.Reducer]\n public static void Crud(ReducerContext ctx)\n {\n ctx.Db.User.Insert(new User { Id = 1, Name = \"Alice\", Age = 30, Active = true });\n ctx.Db.User.Insert(new User { Id = 2, Name = \"Bob\", Age = 22, Active = false });\n\n var found = ctx.Db.User.Id.Find(1);\n if (found is User user)\n {\n user.Name = \"Alice2\";\n user.Age = 31;\n user.Active = false;\n ctx.Db.User.Id.Update(user);\n }\n\n ctx.Db.User.Id.Delete(2);\n }\n}", + "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"User\")]\n public partial struct User\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public string Name;\n public int Age;\n public bool Active;\n }\n\n [SpacetimeDB.Reducer]\n public static void Crud(ReducerContext ctx)\n {\n ctx.Db.User.Insert(new User { Id = 1, Name = \"Alice\", Age = 30, Active = true });\n ctx.Db.User.Insert(new User { Id = 2, Name = \"Bob\", Age = 22, Active = false });\n\n if (ctx.Db.User.Id.Find(1) is User u)\n {\n u.Name = \"Alice2\";\n u.Age = 31;\n u.Active = false;\n ctx.Db.User.Id.Update(u);\n }\n\n ctx.Db.User.Id.Delete(2);\n }\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-007-crud-golden", @@ -2617,20 +2685,11 @@ "llm_db": "basics-t-007-crud-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:34379", + "server": "http://127.0.0.1:42031", "tables_diff": null, "tables_equal": true } }, - "crud_total_count_one": { - "pass": true, - "partial": 1.0, - "notes": { - "actual": 1, - "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM User" - } - }, "crud_row_id2_deleted": { "pass": true, "partial": 1.0, @@ -2651,23 +2710,41 @@ "llm_out": "Id | Name | Age | Active ----+----------+-----+-------- 1 | \"Alice2\" | 31 | false", "query": "SELECT Id, Name, Age, Active FROM User WHERE Id=1", "reducer": "Crud", - "server": "http://127.0.0.1:34379" + "server": "http://127.0.0.1:42031" + } + }, + "crud_total_count_one": { + "pass": true, + "partial": 1.0, + "notes": { + "actual": 1, + "expected": 1, + "sql": "SELECT COUNT(*) AS n FROM User" + } + }, + "crud_row_id2_deleted": { + "pass": true, + "partial": 1.0, + "notes": { + "actual": 0, + "expected": 0, + "sql": "SELECT COUNT(*) AS n FROM User WHERE Id=2" } } }, "vendor": "openai", - "started_at": "2026-01-27T19:59:29.414134061Z", - "finished_at": "2026-01-27T20:00:06.267814536Z" + "started_at": "2026-01-27T22:06:22.563944298Z", + "finished_at": "2026-01-27T22:07:08.425768578Z" }, "t_008_index_lookup": { - "hash": "e3621eaacfd5238fb514ff8c09c277ccd66170773959e5b391507110bf3ac17b", + "hash": "bd31092a0af7d25ac3ccd5ede135c9840ed8cad3c2d6006ee1ad0f5f69e3a3e1", "task": "t_008_index_lookup", "lang": "csharp", "golden_published": true, "model_name": "GPT-5", "total_tests": 3, "passed_tests": 3, - "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"User\")]\n public partial class User\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public string Name;\n public int Age;\n public bool Active;\n }\n\n [SpacetimeDB.Table(Name = \"Result\")]\n public partial class Result\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public string Name;\n }\n\n [SpacetimeDB.Reducer]\n public static void LookupUserName(ReducerContext ctx, int id)\n {\n var user = ctx.Db.User.Id.Find(id);\n if (user != null)\n {\n ctx.Db.Result.Insert(new Result { Id = user.Id, Name = user.Name });\n }\n }\n}", + "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"User\", Public = true)]\n public partial struct User\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public string Name;\n public int Age;\n public bool Active;\n }\n\n [SpacetimeDB.Table(Name = \"Result\", Public = true)]\n public partial struct Result\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public string Name;\n }\n\n [SpacetimeDB.Reducer]\n public static void LookupUserName(ReducerContext ctx, int id)\n {\n if (ctx.Db.User.Id.Find(id) is User user)\n {\n ctx.Db.Result.Insert(new Result\n {\n Id = user.Id,\n Name = user.Name\n });\n }\n }\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-008-index-lookup-golden", @@ -2675,33 +2752,40 @@ "work_dir_golden": "target/llm-runs/basics/t_008_index_lookup/csharp/server/golden", "work_dir_llm": "target/llm-runs/basics/t_008_index_lookup/csharp/server/gpt-5/llm", "scorer_details": { - "index_lookup_projection_parity": { + "schema_parity": { "pass": true, "partial": 1.0, "notes": { - "args": [ - 1 - ], "golden_db": "basics-t-008-index-lookup-golden", - "golden_out": "Id | Name ----+--------- 1 | \"Alice\"", "llm_db": "basics-t-008-index-lookup-gpt-5-llm", - "llm_out": "Id | Name ----+--------- 1 | \"Alice\"", - "query": "SELECT Id, Name FROM Result WHERE Id=1", - "reducer": "LookupUserName", - "server": "http://127.0.0.1:34379" + "reducers_diff": null, + "reducers_equal": true, + "server": "http://127.0.0.1:42031", + "tables_diff": null, + "tables_equal": true } }, - "schema_parity": { + "seed_user_row": { "pass": true, "partial": 1.0, "notes": { + "sql": "INSERT INTO User(Id, Name, Age, Active) VALUES (1, 'Alice', 30, true)" + } + }, + "index_lookup_projection_parity": { + "pass": true, + "partial": 1.0, + "notes": { + "args": [ + 1 + ], "golden_db": "basics-t-008-index-lookup-golden", + "golden_out": "Id | Name ----+--------- 1 | \"Alice\"", "llm_db": "basics-t-008-index-lookup-gpt-5-llm", - "reducers_diff": null, - "reducers_equal": true, - "server": "http://127.0.0.1:34379", - "tables_diff": null, - "tables_equal": true + "llm_out": "Id | Name ----+--------- 1 | \"Alice\"", + "query": "SELECT Id, Name FROM Result WHERE Id=1", + "reducer": "LookupUserName", + "server": "http://127.0.0.1:42031" } }, "seed_user_row": { @@ -2713,11 +2797,11 @@ } }, "vendor": "openai", - "started_at": "2026-01-27T20:00:17.332669093Z", - "finished_at": "2026-01-27T20:00:55.597420674Z" + "started_at": "2026-01-27T22:06:18.642783143Z", + "finished_at": "2026-01-27T22:07:03.442207582Z" }, "t_009_init": { - "hash": "e3621eaacfd5238fb514ff8c09c277ccd66170773959e5b391507110bf3ac17b", + "hash": "bd31092a0af7d25ac3ccd5ede135c9840ed8cad3c2d6006ee1ad0f5f69e3a3e1", "task": "t_009_init", "lang": "csharp", "golden_published": true, @@ -2732,35 +2816,22 @@ "work_dir_golden": "target/llm-runs/basics/t_009_init/csharp/server/golden", "work_dir_llm": "target/llm-runs/basics/t_009_init/csharp/server/gpt-5/llm", "scorer_details": { - "schema_parity": { - "pass": true, - "partial": 1.0, - "notes": { - "golden_db": "basics-t-009-init-golden", - "llm_db": "basics-t-009-init-gpt-5-llm", - "reducers_diff": null, - "reducers_equal": true, - "server": "http://127.0.0.1:34379", - "tables_diff": null, - "tables_equal": true - } - }, - "init_seed_bob": { + "init_seed_alice": { "pass": true, "partial": 1.0, "notes": { "actual": 1, "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM User WHERE Id=2 AND Name='Bob' AND Age=22 AND Active=false" + "sql": "SELECT COUNT(*) AS n FROM User WHERE Id=1 AND Name='Alice' AND Age=30 AND Active=true" } }, - "init_seed_alice": { + "init_total_two": { "pass": true, "partial": 1.0, "notes": { - "actual": 1, - "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM User WHERE Id=1 AND Name='Alice' AND Age=30 AND Active=true" + "actual": 2, + "expected": 2, + "sql": "SELECT COUNT(*) AS n FROM User" } }, "init_total_two": { @@ -2771,14 +2842,27 @@ "expected": 2, "sql": "SELECT COUNT(*) AS n FROM User" } + }, + "schema_parity": { + "pass": true, + "partial": 1.0, + "notes": { + "golden_db": "basics-t-009-init-golden", + "llm_db": "basics-t-009-init-gpt-5-llm", + "reducers_diff": null, + "reducers_equal": true, + "server": "http://127.0.0.1:42031", + "tables_diff": null, + "tables_equal": true + } } }, "vendor": "openai", - "started_at": "2026-01-27T19:59:39.007990909Z", - "finished_at": "2026-01-27T20:00:10.594272918Z" + "started_at": "2026-01-27T22:05:50.454149883Z", + "finished_at": "2026-01-27T22:06:22.563910581Z" }, "t_010_connect": { - "hash": "e3621eaacfd5238fb514ff8c09c277ccd66170773959e5b391507110bf3ac17b", + "hash": "bd31092a0af7d25ac3ccd5ede135c9840ed8cad3c2d6006ee1ad0f5f69e3a3e1", "task": "t_010_connect", "lang": "csharp", "golden_published": true, @@ -2801,25 +2885,25 @@ "llm_db": "basics-t-010-connect-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:34379", + "server": "http://127.0.0.1:42031", "tables_diff": null, "tables_equal": true } } }, "vendor": "openai", - "started_at": "2026-01-27T19:59:08.068058110Z", - "finished_at": "2026-01-27T19:59:45.341706421Z" + "started_at": "2026-01-27T22:06:15.967964050Z", + "finished_at": "2026-01-27T22:06:56.852187767Z" }, "t_011_helper_function": { - "hash": "e3621eaacfd5238fb514ff8c09c277ccd66170773959e5b391507110bf3ac17b", + "hash": "bd31092a0af7d25ac3ccd5ede135c9840ed8cad3c2d6006ee1ad0f5f69e3a3e1", "task": "t_011_helper_function", "lang": "csharp", "golden_published": true, "model_name": "GPT-5", "total_tests": 3, "passed_tests": 3, - "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"Result\")]\n public partial struct Result\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public int Sum;\n }\n\n public static int Add(int a, int b)\n {\n return a + b;\n }\n\n [SpacetimeDB.Reducer]\n public static void ComputeSum(ReducerContext ctx, int id, int a, int b)\n {\n ctx.Db.Result.Insert(new Result\n {\n Id = id,\n Sum = Add(a, b),\n });\n }\n}", + "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"Result\")]\n public partial struct Result\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public int Sum;\n }\n\n private static int Add(int a, int b)\n {\n return a + b;\n }\n\n [SpacetimeDB.Reducer]\n public static void ComputeSum(ReducerContext ctx, int id, int a, int b)\n {\n ctx.Db.Result.Insert(new Result { Id = id, Sum = Add(a, b) });\n }\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-011-helper-function-golden", @@ -2831,9 +2915,13 @@ "pass": true, "partial": 1.0, "notes": { - "actual": 1, - "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM Result WHERE Id=1 AND Sum=5" + "golden_db": "basics-t-011-helper-function-golden", + "llm_db": "basics-t-011-helper-function-gpt-5-llm", + "reducers_diff": null, + "reducers_equal": true, + "server": "http://127.0.0.1:42031", + "tables_diff": null, + "tables_equal": true } }, "helper_func_sum_parity": { @@ -2851,7 +2939,7 @@ "llm_out": "Id | Sum ----+----- 1 | 5", "query": "SELECT Id, Sum FROM Result WHERE Id=1", "reducer": "ComputeSum", - "server": "http://127.0.0.1:34379" + "server": "http://127.0.0.1:42031" } }, "schema_parity": { @@ -2869,11 +2957,11 @@ } }, "vendor": "openai", - "started_at": "2026-01-27T20:00:37.824007777Z", - "finished_at": "2026-01-27T20:01:12.698364118Z" + "started_at": "2026-01-27T22:05:47.405807074Z", + "finished_at": "2026-01-27T22:06:18.642743657Z" }, "t_012_spacetime_product_type": { - "hash": "e3621eaacfd5238fb514ff8c09c277ccd66170773959e5b391507110bf3ac17b", + "hash": "bd31092a0af7d25ac3ccd5ede135c9840ed8cad3c2d6006ee1ad0f5f69e3a3e1", "task": "t_012_spacetime_product_type", "lang": "csharp", "golden_published": true, @@ -2903,7 +2991,16 @@ "llm_out": "Id | Value ----+----------------------- 1 | (Left = 2, Right = 3)", "query": "SELECT Id, Value FROM Result WHERE Id=1", "reducer": "SetScore", - "server": "http://127.0.0.1:34379" + "server": "http://127.0.0.1:42031" + } + }, + "product_type_row_count": { + "pass": true, + "partial": 1.0, + "notes": { + "actual": 1, + "expected": 1, + "sql": "SELECT COUNT(*) AS n FROM Result WHERE Id=1" } }, "schema_parity": { @@ -2914,7 +3011,7 @@ "llm_db": "schema-t-012-spacetime-product-type-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:34379", + "server": "http://127.0.0.1:42031", "tables_diff": null, "tables_equal": true } @@ -2930,18 +3027,18 @@ } }, "vendor": "openai", - "started_at": "2026-01-27T19:58:50.225754898Z", - "finished_at": "2026-01-27T19:59:29.413988946Z" + "started_at": "2026-01-27T22:02:31.393119815Z", + "finished_at": "2026-01-27T22:03:52.200736367Z" }, "t_013_spacetime_sum_type": { - "hash": "e3621eaacfd5238fb514ff8c09c277ccd66170773959e5b391507110bf3ac17b", + "hash": "bd31092a0af7d25ac3ccd5ede135c9840ed8cad3c2d6006ee1ad0f5f69e3a3e1", "task": "t_013_spacetime_sum_type", "lang": "csharp", "golden_published": true, "model_name": "GPT-5", "total_tests": 3, "passed_tests": 3, - "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Type]\n public partial struct Circle\n {\n public int Radius;\n }\n\n [SpacetimeDB.Type]\n public partial struct Rectangle\n {\n public int Width;\n public int Height;\n }\n\n [SpacetimeDB.Type]\n public partial record Shape : TaggedEnum<(Circle Circle, Rectangle Rectangle)> { }\n\n [SpacetimeDB.Table(Name = \"Result\")]\n public partial struct Result\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public Shape Value;\n }\n\n [SpacetimeDB.Reducer]\n public static void SetCircle(ReducerContext ctx, int id, int radius)\n {\n ctx.Db.Result.Insert(new Result\n {\n Id = id,\n Value = new Shape.Circle(new Circle { Radius = radius })\n });\n }\n}", + "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Type]\n public partial struct Circle\n {\n public int Radius;\n }\n\n [SpacetimeDB.Type]\n public partial struct Rectangle\n {\n public int Width;\n public int Height;\n }\n\n [SpacetimeDB.Type]\n public partial record Shape : TaggedEnum<(Circle Circle, Rectangle Rectangle)> { }\n\n [SpacetimeDB.Table(Name = \"Result\")]\n public partial struct Result\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public Shape Value;\n }\n\n [SpacetimeDB.Reducer]\n public static void SetCircle(ReducerContext ctx, int id, int radius)\n {\n var circle = new Circle { Radius = radius };\n var shape = new Shape.Circle(circle);\n ctx.Db.Result.Insert(new Result { Id = id, Value = shape });\n }\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-013-spacetime-sum-type-golden", @@ -2949,6 +3046,23 @@ "work_dir_golden": "target/llm-runs/schema/t_013_spacetime_sum_type/csharp/server/golden", "work_dir_llm": "target/llm-runs/schema/t_013_spacetime_sum_type/csharp/server/gpt-5/llm", "scorer_details": { + "sum_type_row_parity": { + "pass": true, + "partial": 1.0, + "notes": { + "args": [ + 1, + 10 + ], + "golden_db": "schema-t-013-spacetime-sum-type-golden", + "golden_out": "Id | Value ----+-------------------------- 1 | (Circle = (Radius = 10))", + "llm_db": "schema-t-013-spacetime-sum-type-gpt-5-llm", + "llm_out": "Id | Value ----+-------------------------- 1 | (Circle = (Radius = 10))", + "query": "SELECT Id, Value FROM Result WHERE Id=1", + "reducer": "SetCircle", + "server": "http://127.0.0.1:42031" + } + }, "schema_parity": { "pass": true, "partial": 1.0, @@ -2957,7 +3071,7 @@ "llm_db": "schema-t-013-spacetime-sum-type-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:34379", + "server": "http://127.0.0.1:42031", "tables_diff": null, "tables_equal": true } @@ -2970,31 +3084,14 @@ "expected": 1, "sql": "SELECT COUNT(*) AS n FROM Result WHERE Id=1" } - }, - "sum_type_row_parity": { - "pass": true, - "partial": 1.0, - "notes": { - "args": [ - 1, - 10 - ], - "golden_db": "schema-t-013-spacetime-sum-type-golden", - "golden_out": "Id | Value ----+-------------------------- 1 | (Circle = (Radius = 10))", - "llm_db": "schema-t-013-spacetime-sum-type-gpt-5-llm", - "llm_out": "Id | Value ----+-------------------------- 1 | (Circle = (Radius = 10))", - "query": "SELECT Id, Value FROM Result WHERE Id=1", - "reducer": "SetCircle", - "server": "http://127.0.0.1:34379" - } } }, "vendor": "openai", - "started_at": "2026-01-27T19:57:55.737450825Z", - "finished_at": "2026-01-27T19:58:46.370198569Z" + "started_at": "2026-01-27T22:02:31.395674019Z", + "finished_at": "2026-01-27T22:04:09.953769907Z" }, "t_014_elementary_columns": { - "hash": "e3621eaacfd5238fb514ff8c09c277ccd66170773959e5b391507110bf3ac17b", + "hash": "bd31092a0af7d25ac3ccd5ede135c9840ed8cad3c2d6006ee1ad0f5f69e3a3e1", "task": "t_014_elementary_columns", "lang": "csharp", "golden_published": true, @@ -3009,49 +3106,49 @@ "work_dir_golden": "target/llm-runs/schema/t_014_elementary_columns/csharp/server/golden", "work_dir_llm": "target/llm-runs/schema/t_014_elementary_columns/csharp/server/gpt-5/llm", "scorer_details": { - "schema_parity": { - "pass": true, - "partial": 1.0, + "elementary_columns_row_parity": { + "pass": false, + "partial": 0.0, "notes": { - "golden_db": "schema-t-014-elementary-columns-golden", - "llm_db": "schema-t-014-elementary-columns-gpt-5-llm", - "reducers_diff": null, - "reducers_equal": true, - "server": "http://127.0.0.1:34379", - "tables_diff": null, - "tables_equal": true + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `primitive`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:42031/v1/database/c20043b845b8085a07c0b477548855329e4796100f0f03364d36a179a90383cc/sql)\n", + "phase": "sql_golden" } }, "elementary_columns_row_count": { "pass": false, "partial": 0.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `primitive`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:34379/v1/database/c2006db3efe118f9e2be4d0996245799216709f62c09af27307321501d4ca96f/sql)\n", + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `primitive`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:42031/v1/database/c2007013a11da860089b5483efb500927c3a045571f20e8c11d044d87f4af1db/sql)\n", "phase": "sql" } }, - "elementary_columns_row_parity": { - "pass": false, - "partial": 0.0, + "schema_parity": { + "pass": true, + "partial": 1.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `primitive`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:34379/v1/database/c2008bda321ec5b24f06db6690e57eaa2c1b3a9bada2318accf99feb7910837d/sql)\n", - "phase": "sql_golden" + "golden_db": "schema-t-014-elementary-columns-golden", + "llm_db": "schema-t-014-elementary-columns-gpt-5-llm", + "reducers_diff": null, + "reducers_equal": true, + "server": "http://127.0.0.1:42031", + "tables_diff": null, + "tables_equal": true } } }, "vendor": "openai", - "started_at": "2026-01-27T19:58:46.370277696Z", - "finished_at": "2026-01-27T19:59:43.681502253Z" + "started_at": "2026-01-27T22:04:26.310123052Z", + "finished_at": "2026-01-27T22:05:00.112608530Z" }, "t_015_product_type_columns": { - "hash": "e3621eaacfd5238fb514ff8c09c277ccd66170773959e5b391507110bf3ac17b", + "hash": "bd31092a0af7d25ac3ccd5ede135c9840ed8cad3c2d6006ee1ad0f5f69e3a3e1", "task": "t_015_product_type_columns", "lang": "csharp", "golden_published": true, "model_name": "GPT-5", "total_tests": 3, "passed_tests": 3, - "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Type]\n public partial struct Address\n {\n public string Street;\n public int Zip;\n }\n\n [SpacetimeDB.Type]\n public partial struct Position\n {\n public int X;\n public int Y;\n }\n\n [SpacetimeDB.Table(Name = \"Profile\", Public = true)]\n public partial struct Profile\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public Address Home;\n public Address Work;\n public Position Pos;\n }\n\n [SpacetimeDB.Reducer]\n public static void Seed(ReducerContext ctx)\n {\n if (ctx.Db.Profile.Id.Find(1) == null)\n {\n ctx.Db.Profile.Insert(new Profile\n {\n Id = 1,\n Home = new Address { Street = \"1 Main\", Zip = 11111 },\n Work = new Address { Street = \"2 Broad\", Zip = 22222 },\n Pos = new Position { X = 7, Y = 9 },\n });\n }\n }\n}", + "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Type]\n public partial struct Address\n {\n public string Street;\n public int Zip;\n }\n\n [SpacetimeDB.Type]\n public partial struct Position\n {\n public int X;\n public int Y;\n }\n\n [SpacetimeDB.Table(Name = \"Profile\", Public = true)]\n public partial struct Profile\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public Address Home;\n public Address Work;\n public Position Pos;\n }\n\n [SpacetimeDB.Reducer]\n public static void Seed(ReducerContext ctx)\n {\n ctx.Db.Profile.Insert(new Profile\n {\n Id = 1,\n Home = new Address { Street = \"1 Main\", Zip = 11111 },\n Work = new Address { Street = \"2 Broad\", Zip = 22222 },\n Pos = new Position { X = 7, Y = 9 }\n });\n }\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-015-product-type-columns-golden", @@ -3059,31 +3156,31 @@ "work_dir_golden": "target/llm-runs/schema/t_015_product_type_columns/csharp/server/golden", "work_dir_llm": "target/llm-runs/schema/t_015_product_type_columns/csharp/server/gpt-5/llm", "scorer_details": { - "schema_parity": { + "product_type_columns_row_parity": { "pass": true, "partial": 1.0, "notes": { + "args": [], "golden_db": "schema-t-015-product-type-columns-golden", + "golden_out": "Id | Home | Work | Pos ----+----------------------------------+-----------------------------------+---------------- 1 | (Street = \"1 Main\", Zip = 11111) | (Street = \"2 Broad\", Zip = 22222) | (X = 7, Y = 9)", "llm_db": "schema-t-015-product-type-columns-gpt-5-llm", - "reducers_diff": null, - "reducers_equal": true, - "server": "http://127.0.0.1:34379", - "tables_diff": null, - "tables_equal": true + "llm_out": "Id | Home | Work | Pos ----+----------------------------------+-----------------------------------+---------------- 1 | (Street = \"1 Main\", Zip = 11111) | (Street = \"2 Broad\", Zip = 22222) | (X = 7, Y = 9)", + "query": "SELECT Id, Home, Work, Pos FROM Profile WHERE Id=1", + "reducer": "Seed", + "server": "http://127.0.0.1:42031" } }, - "product_type_columns_row_parity": { + "product_type_columns_row_count": { "pass": true, "partial": 1.0, "notes": { - "args": [], "golden_db": "schema-t-015-product-type-columns-golden", - "golden_out": "Id | Home | Work | Pos ----+----------------------------------+-----------------------------------+---------------- 1 | (Street = \"1 Main\", Zip = 11111) | (Street = \"2 Broad\", Zip = 22222) | (X = 7, Y = 9)", "llm_db": "schema-t-015-product-type-columns-gpt-5-llm", - "llm_out": "Id | Home | Work | Pos ----+----------------------------------+-----------------------------------+---------------- 1 | (Street = \"1 Main\", Zip = 11111) | (Street = \"2 Broad\", Zip = 22222) | (X = 7, Y = 9)", - "query": "SELECT Id, Home, Work, Pos FROM Profile WHERE Id=1", - "reducer": "Seed", - "server": "http://127.0.0.1:34379" + "reducers_diff": null, + "reducers_equal": true, + "server": "http://127.0.0.1:42031", + "tables_diff": null, + "tables_equal": true } }, "product_type_columns_row_count": { @@ -3097,11 +3194,11 @@ } }, "vendor": "openai", - "started_at": "2026-01-27T19:58:04.285909374Z", - "finished_at": "2026-01-27T19:58:50.225511855Z" + "started_at": "2026-01-27T22:03:56.262251047Z", + "finished_at": "2026-01-27T22:04:31.236049938Z" }, "t_016_sum_type_columns": { - "hash": "e3621eaacfd5238fb514ff8c09c277ccd66170773959e5b391507110bf3ac17b", + "hash": "bd31092a0af7d25ac3ccd5ede135c9840ed8cad3c2d6006ee1ad0f5f69e3a3e1", "task": "t_016_sum_type_columns", "lang": "csharp", "golden_published": true, @@ -3120,10 +3217,18 @@ "pass": false, "partial": 0.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `drawings`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:34379/v1/database/c200a38bd5eaec739ca1bb5f49423cf455fb484e997f330a37dbe10a2f09faac/sql)\n", + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `drawings`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:42031/v1/database/c20070cd03a56bffcc38458dcb1029b1991f9498b6a12c9a46a7878bffe10d49/sql)\n", "phase": "sql" } }, + "sum_type_columns_row_parity": { + "pass": false, + "partial": 0.0, + "notes": { + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `drawings`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:42031/v1/database/c200eaa8f05c885ac60455478b9b68d6b18bb1b92f299c08c43cdd29ef1b9a97/sql)\n", + "phase": "sql_golden" + } + }, "schema_parity": { "pass": true, "partial": 1.0, @@ -3132,33 +3237,25 @@ "llm_db": "schema-t-016-sum-type-columns-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:34379", + "server": "http://127.0.0.1:42031", "tables_diff": null, "tables_equal": true } - }, - "sum_type_columns_row_parity": { - "pass": false, - "partial": 0.0, - "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `drawings`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:34379/v1/database/c20028da21a53491f84d8895dbcf4906c6c844846426c39ae32b5fd1689a2641/sql)\n", - "phase": "sql_golden" - } } }, "vendor": "openai", - "started_at": "2026-01-27T19:57:09.422528630Z", - "finished_at": "2026-01-27T19:58:04.281919840Z" + "started_at": "2026-01-27T22:02:31.398828215Z", + "finished_at": "2026-01-27T22:03:56.262197561Z" }, "t_017_scheduled_columns": { - "hash": "e3621eaacfd5238fb514ff8c09c277ccd66170773959e5b391507110bf3ac17b", + "hash": "bd31092a0af7d25ac3ccd5ede135c9840ed8cad3c2d6006ee1ad0f5f69e3a3e1", "task": "t_017_scheduled_columns", "lang": "csharp", "golden_published": true, "model_name": "GPT-5", "total_tests": 2, "passed_tests": 1, - "llm_output": "using System;\nusing SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"TickTimer\", Scheduled = nameof(Tick), ScheduledAt = nameof(ScheduledAt))]\n public partial struct TickTimer\n {\n [SpacetimeDB.PrimaryKey]\n [SpacetimeDB.AutoInc]\n public ulong ScheduledId;\n public ScheduleAt ScheduledAt;\n }\n\n [SpacetimeDB.Reducer]\n public static void Tick(ReducerContext ctx, TickTimer _timer)\n {\n }\n\n [SpacetimeDB.Reducer(ReducerKind.Init)]\n public static void Init(ReducerContext ctx)\n {\n ctx.Db.TickTimer.Insert(new TickTimer\n {\n ScheduledAt = new ScheduleAt.Interval(TimeSpan.FromMilliseconds(50))\n });\n }\n}", + "llm_output": "using System;\nusing SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"TickTimer\", Scheduled = nameof(Tick), ScheduledAt = nameof(ScheduledAt))]\n public partial struct TickTimer\n {\n [SpacetimeDB.PrimaryKey]\n [SpacetimeDB.AutoInc]\n public ulong ScheduledId;\n public ScheduleAt ScheduledAt;\n }\n\n [SpacetimeDB.Reducer(SpacetimeDB.ReducerKind.Init)]\n public static void Init(ReducerContext ctx)\n {\n ctx.Db.TickTimer.Insert(new TickTimer\n {\n ScheduledAt = new ScheduleAt.Interval(TimeSpan.FromMilliseconds(50))\n });\n }\n\n [SpacetimeDB.Reducer]\n public static void Tick(ReducerContext ctx, TickTimer TickTimer)\n {\n }\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-017-scheduled-columns-golden", @@ -3182,18 +3279,26 @@ "llm_db": "schema-t-017-scheduled-columns-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:34379", + "server": "http://127.0.0.1:42031", "tables_diff": null, "tables_equal": true } + }, + "scheduled_seeded_one_row": { + "pass": false, + "partial": 0.0, + "notes": { + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `tick_timer`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:42031/v1/database/c200e44961ff20a44bd250567c5dcaf9c11e2455c3afcfdfd773be3c4042003a/sql)\n", + "phase": "sql" + } } }, "vendor": "openai", - "started_at": "2026-01-27T19:57:09.413847953Z", - "finished_at": "2026-01-27T19:57:55.737395054Z" + "started_at": "2026-01-27T22:04:09.953815313Z", + "finished_at": "2026-01-27T22:04:50.875665709Z" }, "t_018_constraints": { - "hash": "e3621eaacfd5238fb514ff8c09c277ccd66170773959e5b391507110bf3ac17b", + "hash": "bd31092a0af7d25ac3ccd5ede135c9840ed8cad3c2d6006ee1ad0f5f69e3a3e1", "task": "t_018_constraints", "lang": "csharp", "golden_published": true, @@ -3208,6 +3313,15 @@ "work_dir_golden": "target/llm-runs/schema/t_018_constraints/csharp/server/golden", "work_dir_llm": "target/llm-runs/schema/t_018_constraints/csharp/server/gpt-5/llm", "scorer_details": { + "constraints_seed_two_rows": { + "pass": true, + "partial": 1.0, + "notes": { + "actual": 1, + "expected": 1, + "sql": "SELECT COUNT(*) AS n FROM Account WHERE Id=2" + } + }, "constraints_row_parity_after_seed": { "pass": true, "partial": 1.0, @@ -3219,16 +3333,7 @@ "llm_out": "Id | Email | Name ----+-----------------+--------- 1 | \"a@example.com\" | \"Alice\"", "query": "SELECT Id, Email, Name FROM Account WHERE Id=1", "reducer": "Seed", - "server": "http://127.0.0.1:34379" - } - }, - "constraints_seed_two_rows": { - "pass": true, - "partial": 1.0, - "notes": { - "actual": 1, - "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM Account WHERE Id=2" + "server": "http://127.0.0.1:42031" } }, "schema_parity": { @@ -3239,25 +3344,25 @@ "llm_db": "schema-t-018-constraints-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:34379", + "server": "http://127.0.0.1:42031", "tables_diff": null, "tables_equal": true } } }, "vendor": "openai", - "started_at": "2026-01-27T19:57:09.410769526Z", - "finished_at": "2026-01-27T19:58:03.584746288Z" + "started_at": "2026-01-27T22:02:31.402276922Z", + "finished_at": "2026-01-27T22:03:52.031753547Z" }, "t_019_many_to_many": { - "hash": "e3621eaacfd5238fb514ff8c09c277ccd66170773959e5b391507110bf3ac17b", + "hash": "bd31092a0af7d25ac3ccd5ede135c9840ed8cad3c2d6006ee1ad0f5f69e3a3e1", "task": "t_019_many_to_many", "lang": "csharp", "golden_published": true, "model_name": "GPT-5", "total_tests": 5, "passed_tests": 5, - "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"User\")]\n public partial struct User\n {\n [SpacetimeDB.PrimaryKey]\n public int UserId;\n public string Name;\n }\n\n [SpacetimeDB.Table(Name = \"Group\")]\n public partial struct Group\n {\n [SpacetimeDB.PrimaryKey]\n public int GroupId;\n public string Title;\n }\n\n [SpacetimeDB.Table(Name = \"Membership\")]\n [SpacetimeDB.Index.BTree(Name = \"by_user\", Columns = new[] { \"UserId\" })]\n [SpacetimeDB.Index.BTree(Name = \"by_group\", Columns = new[] { \"GroupId\" })]\n public partial struct Membership\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public int UserId;\n public int GroupId;\n }\n\n [SpacetimeDB.Reducer]\n public static void Seed(ReducerContext ctx)\n {\n foreach (var m in ctx.Db.Membership.Iter())\n {\n ctx.Db.Membership.Id.Delete(m.Id);\n }\n\n foreach (var g in ctx.Db.Group.Iter())\n {\n ctx.Db.Group.GroupId.Delete(g.GroupId);\n }\n\n foreach (var u in ctx.Db.User.Iter())\n {\n ctx.Db.User.UserId.Delete(u.UserId);\n }\n\n ctx.Db.User.Insert(new User { UserId = 1, Name = \"Alice\" });\n ctx.Db.User.Insert(new User { UserId = 2, Name = \"Bob\" });\n\n ctx.Db.Group.Insert(new Group { GroupId = 10, Title = \"Admin\" });\n ctx.Db.Group.Insert(new Group { GroupId = 20, Title = \"Dev\" });\n\n ctx.Db.Membership.Insert(new Membership { Id = 1, UserId = 1, GroupId = 10 });\n ctx.Db.Membership.Insert(new Membership { Id = 2, UserId = 1, GroupId = 20 });\n ctx.Db.Membership.Insert(new Membership { Id = 3, UserId = 2, GroupId = 20 });\n }\n}", + "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"User\", Public = true)]\n public partial struct User\n {\n [SpacetimeDB.PrimaryKey]\n public int UserId;\n public string Name;\n }\n\n [SpacetimeDB.Table(Name = \"Group\", Public = true)]\n public partial struct Group\n {\n [SpacetimeDB.PrimaryKey]\n public int GroupId;\n public string Title;\n }\n\n [SpacetimeDB.Table(Name = \"Membership\", Public = true)]\n [SpacetimeDB.Index.BTree(Name = \"by_user\", Columns = new[] { nameof(UserId) })]\n [SpacetimeDB.Index.BTree(Name = \"by_group\", Columns = new[] { nameof(GroupId) })]\n public partial struct Membership\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public int UserId;\n public int GroupId;\n }\n\n [SpacetimeDB.Reducer]\n public static void Seed(ReducerContext ctx)\n {\n foreach (var m in ctx.Db.Membership.Iter())\n {\n ctx.Db.Membership.Id.Delete(m.Id);\n }\n foreach (var g in ctx.Db.Group.Iter())\n {\n ctx.Db.Group.GroupId.Delete(g.GroupId);\n }\n foreach (var u in ctx.Db.User.Iter())\n {\n ctx.Db.User.UserId.Delete(u.UserId);\n }\n\n ctx.Db.User.Insert(new User { UserId = 1, Name = \"Alice\" });\n ctx.Db.User.Insert(new User { UserId = 2, Name = \"Bob\" });\n\n ctx.Db.Group.Insert(new Group { GroupId = 10, Title = \"Admin\" });\n ctx.Db.Group.Insert(new Group { GroupId = 20, Title = \"Dev\" });\n\n ctx.Db.Membership.Insert(new Membership { Id = 1, UserId = 1, GroupId = 10 });\n ctx.Db.Membership.Insert(new Membership { Id = 2, UserId = 1, GroupId = 20 });\n ctx.Db.Membership.Insert(new Membership { Id = 3, UserId = 2, GroupId = 20 });\n }\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-019-many-to-many-golden", @@ -3265,69 +3370,69 @@ "work_dir_golden": "target/llm-runs/schema/t_019_many_to_many/csharp/server/golden", "work_dir_llm": "target/llm-runs/schema/t_019_many_to_many/csharp/server/gpt-5/llm", "scorer_details": { - "m2m_has_2_20": { + "m2m_has_1_20": { "pass": true, "partial": 1.0, "notes": { "actual": 1, "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM Membership WHERE UserId=2 AND GroupId=20" + "sql": "SELECT COUNT(*) AS n FROM Membership WHERE UserId=1 AND GroupId=20" } }, - "memberships_three_rows": { + "m2m_has_1_10": { "pass": true, "partial": 1.0, "notes": { - "actual": 3, - "expected": 3, - "sql": "SELECT COUNT(*) AS n FROM Membership" + "actual": 1, + "expected": 1, + "sql": "SELECT COUNT(*) AS n FROM Membership WHERE UserId=1 AND GroupId=10" } }, - "schema_parity": { + "memberships_three_rows": { "pass": true, "partial": 1.0, "notes": { - "golden_db": "schema-t-019-many-to-many-golden", - "llm_db": "schema-t-019-many-to-many-gpt-5-llm", - "reducers_diff": null, - "reducers_equal": true, - "server": "http://127.0.0.1:34379", - "tables_diff": null, - "tables_equal": true + "actual": 3, + "expected": 3, + "sql": "SELECT COUNT(*) AS n FROM Membership" } }, - "m2m_has_1_20": { + "m2m_has_2_20": { "pass": true, "partial": 1.0, "notes": { "actual": 1, "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM Membership WHERE UserId=1 AND GroupId=20" + "sql": "SELECT COUNT(*) AS n FROM Membership WHERE UserId=2 AND GroupId=20" } }, - "m2m_has_1_10": { + "schema_parity": { "pass": true, "partial": 1.0, "notes": { - "actual": 1, - "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM Membership WHERE UserId=1 AND GroupId=10" + "golden_db": "schema-t-019-many-to-many-golden", + "llm_db": "schema-t-019-many-to-many-gpt-5-llm", + "reducers_diff": null, + "reducers_equal": true, + "server": "http://127.0.0.1:42031", + "tables_diff": null, + "tables_equal": true } } }, "vendor": "openai", - "started_at": "2026-01-27T19:58:04.046511986Z", - "finished_at": "2026-01-27T19:58:58.263921552Z" + "started_at": "2026-01-27T22:04:31.236092692Z", + "finished_at": "2026-01-27T22:05:42.457644003Z" }, "t_020_ecs": { - "hash": "e3621eaacfd5238fb514ff8c09c277ccd66170773959e5b391507110bf3ac17b", + "hash": "bd31092a0af7d25ac3ccd5ede135c9840ed8cad3c2d6006ee1ad0f5f69e3a3e1", "task": "t_020_ecs", "lang": "csharp", "golden_published": true, "model_name": "GPT-5", "total_tests": 5, "passed_tests": 1, - "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"Entity\")]\n public partial struct Entity\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n }\n\n [SpacetimeDB.Table(Name = \"Position\")]\n public partial struct Position\n {\n [SpacetimeDB.PrimaryKey]\n public int EntityId;\n public int X;\n public int Y;\n }\n\n [SpacetimeDB.Table(Name = \"Velocity\")]\n public partial struct Velocity\n {\n [SpacetimeDB.PrimaryKey]\n public int EntityId;\n public int VX;\n public int VY;\n }\n\n [SpacetimeDB.Table(Name = \"NextPosition\")]\n public partial struct NextPosition\n {\n [SpacetimeDB.PrimaryKey]\n public int EntityId;\n public int X;\n public int Y;\n }\n\n [SpacetimeDB.Reducer]\n public static void Seed(ReducerContext ctx)\n {\n // Entities\n ctx.Db.Entity.Insert(new Entity { Id = 1 });\n ctx.Db.Entity.Insert(new Entity { Id = 2 });\n\n // Positions\n ctx.Db.Position.Insert(new Position { EntityId = 1, X = 0, Y = 0 });\n ctx.Db.Position.Insert(new Position { EntityId = 2, X = 10, Y = 0 });\n\n // Velocities\n ctx.Db.Velocity.Insert(new Velocity { EntityId = 1, VX = 1, VY = 0 });\n ctx.Db.Velocity.Insert(new Velocity { EntityId = 2, VX = -2, VY = 3 });\n }\n\n [SpacetimeDB.Reducer]\n public static void Step(ReducerContext ctx)\n {\n foreach (var pos in ctx.Db.Position.Iter())\n {\n var vel = ctx.Db.Velocity.EntityId.Find(pos.EntityId);\n int vx = 0, vy = 0;\n if (vel != null)\n {\n vx = vel.Value.VX;\n vy = vel.Value.VY;\n }\n\n var next = new NextPosition\n {\n EntityId = pos.EntityId,\n X = pos.X + vx,\n Y = pos.Y + vy\n };\n\n var existing = ctx.Db.NextPosition.EntityId.Find(pos.EntityId);\n if (existing != null)\n {\n ctx.Db.NextPosition.EntityId.Update(next);\n }\n else\n {\n ctx.Db.NextPosition.Insert(next);\n }\n }\n }\n}", + "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"Entity\", Public = true)]\n public partial struct Entity\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n }\n\n [SpacetimeDB.Table(Name = \"Position\", Public = true)]\n public partial struct Position\n {\n [SpacetimeDB.PrimaryKey]\n public int EntityId;\n public int X;\n public int Y;\n }\n\n [SpacetimeDB.Table(Name = \"Velocity\", Public = true)]\n public partial struct Velocity\n {\n [SpacetimeDB.PrimaryKey]\n public int EntityId;\n public int VX;\n public int VY;\n }\n\n [SpacetimeDB.Table(Name = \"NextPosition\", Public = true)]\n public partial struct NextPosition\n {\n [SpacetimeDB.PrimaryKey]\n public int EntityId;\n public int X;\n public int Y;\n }\n\n [SpacetimeDB.Reducer]\n public static void Seed(ReducerContext ctx)\n {\n UpsertEntity(ctx, 1);\n UpsertPosition(ctx, 1, 0, 0);\n UpsertVelocity(ctx, 1, 1, 0);\n\n UpsertEntity(ctx, 2);\n UpsertPosition(ctx, 2, 10, 0);\n UpsertVelocity(ctx, 2, -2, 3);\n }\n\n [SpacetimeDB.Reducer]\n public static void Step(ReducerContext ctx)\n {\n foreach (var pos in ctx.Db.Position.Iter())\n {\n var velOpt = ctx.Db.Velocity.EntityId.Find(pos.EntityId);\n if (velOpt == null)\n {\n continue;\n }\n\n var vel = velOpt.Value;\n var nx = pos.X + vel.VX;\n var ny = pos.Y + vel.VY;\n\n var npOpt = ctx.Db.NextPosition.EntityId.Find(pos.EntityId);\n if (npOpt == null)\n {\n ctx.Db.NextPosition.Insert(new NextPosition\n {\n EntityId = pos.EntityId,\n X = nx,\n Y = ny\n });\n }\n else\n {\n var np = npOpt.Value;\n np.X = nx;\n np.Y = ny;\n ctx.Db.NextPosition.EntityId.Update(np);\n }\n }\n }\n\n static void UpsertEntity(ReducerContext ctx, int id)\n {\n var existing = ctx.Db.Entity.Id.Find(id);\n if (existing == null)\n {\n ctx.Db.Entity.Insert(new Entity { Id = id });\n }\n }\n\n static void UpsertPosition(ReducerContext ctx, int entityId, int x, int y)\n {\n var existing = ctx.Db.Position.EntityId.Find(entityId);\n if (existing == null)\n {\n ctx.Db.Position.Insert(new Position { EntityId = entityId, X = x, Y = y });\n }\n else\n {\n var p = existing.Value;\n p.X = x;\n p.Y = y;\n ctx.Db.Position.EntityId.Update(p);\n }\n }\n\n static void UpsertVelocity(ReducerContext ctx, int entityId, int vx, int vy)\n {\n var existing = ctx.Db.Velocity.EntityId.Find(entityId);\n if (existing == null)\n {\n ctx.Db.Velocity.Insert(new Velocity { EntityId = entityId, VX = vx, VY = vy });\n }\n else\n {\n var v = existing.Value;\n v.VX = vx;\n v.VY = vy;\n ctx.Db.Velocity.EntityId.Update(v);\n }\n }\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-020-ecs-golden", @@ -3335,11 +3440,11 @@ "work_dir_golden": "target/llm-runs/schema/t_020_ecs/csharp/server/golden", "work_dir_llm": "target/llm-runs/schema/t_020_ecs/csharp/server/gpt-5/llm", "scorer_details": { - "ecs_step_next_positions_count": { + "ecs_next_pos_entity2": { "pass": false, "partial": 0.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `next_positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:34379/v1/database/c20015874a697116ed39e7f80d0fb9c484518ec19d2cf28ab6bbc9a525d697e0/sql)\n", + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `next_positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:42031/v1/database/c200dcde9ccab700de2ea96db18b1fe51e357b2dca9c454c0c9e2f4d6cd478fa/sql)\n", "phase": "sql" } }, @@ -3347,15 +3452,7 @@ "pass": false, "partial": 0.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `next_positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:34379/v1/database/c20015874a697116ed39e7f80d0fb9c484518ec19d2cf28ab6bbc9a525d697e0/sql)\n", - "phase": "sql" - } - }, - "ecs_next_pos_entity2": { - "pass": false, - "partial": 0.0, - "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `next_positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:34379/v1/database/c20015874a697116ed39e7f80d0fb9c484518ec19d2cf28ab6bbc9a525d697e0/sql)\n", + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `next_positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:42031/v1/database/c200dcde9ccab700de2ea96db18b1fe51e357b2dca9c454c0c9e2f4d6cd478fa/sql)\n", "phase": "sql" } }, @@ -3367,26 +3464,34 @@ "llm_db": "schema-t-020-ecs-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:34379", + "server": "http://127.0.0.1:42031", "tables_diff": null, "tables_equal": true } }, + "ecs_step_next_positions_count": { + "pass": false, + "partial": 0.0, + "notes": { + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `next_positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:42031/v1/database/c200dcde9ccab700de2ea96db18b1fe51e357b2dca9c454c0c9e2f4d6cd478fa/sql)\n", + "phase": "sql" + } + }, "ecs_seed_positions_count": { "pass": false, "partial": 0.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:34379/v1/database/c20015874a697116ed39e7f80d0fb9c484518ec19d2cf28ab6bbc9a525d697e0/sql)\n", + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:42031/v1/database/c200dcde9ccab700de2ea96db18b1fe51e357b2dca9c454c0c9e2f4d6cd478fa/sql)\n", "phase": "sql" } } }, "vendor": "openai", - "started_at": "2026-01-27T19:58:04.281975972Z", - "finished_at": "2026-01-27T19:59:08.067949434Z" + "started_at": "2026-01-27T22:03:52.204029385Z", + "finished_at": "2026-01-27T22:05:21.015673517Z" }, "t_021_multi_column_index": { - "hash": "e3621eaacfd5238fb514ff8c09c277ccd66170773959e5b391507110bf3ac17b", + "hash": "bd31092a0af7d25ac3ccd5ede135c9840ed8cad3c2d6006ee1ad0f5f69e3a3e1", "task": "t_021_multi_column_index", "lang": "csharp", "golden_published": true, @@ -3401,13 +3506,13 @@ "work_dir_golden": "target/llm-runs/schema/t_021_multi_column_index/csharp/server/golden", "work_dir_llm": "target/llm-runs/schema/t_021_multi_column_index/csharp/server/gpt-5/llm", "scorer_details": { - "mcindex_lookup_u7_d2": { + "mcindex_lookup_u7_d1": { "pass": true, "partial": 1.0, "notes": { "actual": 1, "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM Log WHERE UserId=7 AND Day=2" + "sql": "SELECT COUNT(*) AS n FROM Log WHERE UserId=7 AND Day=1" } }, "mcindex_seed_count": { @@ -3419,13 +3524,13 @@ "sql": "SELECT COUNT(*) AS n FROM Log" } }, - "mcindex_lookup_u7_d1": { + "mcindex_lookup_u7_d2": { "pass": true, "partial": 1.0, "notes": { "actual": 1, "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM Log WHERE UserId=7 AND Day=1" + "sql": "SELECT COUNT(*) AS n FROM Log WHERE UserId=7 AND Day=2" } }, "schema_parity": { @@ -3436,15 +3541,15 @@ "llm_db": "schema-t-021-multi-column-index-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:34379", + "server": "http://127.0.0.1:42031", "tables_diff": null, "tables_equal": true } } }, "vendor": "openai", - "started_at": "2026-01-27T19:57:09.418092225Z", - "finished_at": "2026-01-27T19:58:04.046455117Z" + "started_at": "2026-01-27T22:03:52.200781109Z", + "finished_at": "2026-01-27T22:04:26.310079475Z" } } } diff --git a/docs/llms/docs-benchmark-summary.json b/docs/llms/docs-benchmark-summary.json index bf684185512..d5e83b49997 100644 --- a/docs/llms/docs-benchmark-summary.json +++ b/docs/llms/docs-benchmark-summary.json @@ -1,11 +1,11 @@ { "version": 1, - "generated_at": "2026-01-27T20:01:19.767Z", + "generated_at": "2026-01-27T22:07:18.787Z", "by_language": { "csharp": { "modes": { "docs": { - "hash": "e3621eaacfd5238fb514ff8c09c277ccd66170773959e5b391507110bf3ac17b", + "hash": "bd31092a0af7d25ac3ccd5ede135c9840ed8cad3c2d6006ee1ad0f5f69e3a3e1", "models": { "GPT-5": { "categories": { @@ -42,7 +42,7 @@ "rust": { "modes": { "docs": { - "hash": "d1bf0b033db905c668d016082c762eb4b91edd1299f7b8d4bcc6def5398610f9", + "hash": "d86518d2ac099e60eb25705cd317e30df95f4d6a4787a2fe5aae12753b8ab4f7", "models": { "GPT-5": { "categories": { @@ -56,18 +56,18 @@ }, "schema": { "tasks": 10, - "total_tests": 32, + "total_tests": 34, "passed_tests": 8, - "pass_pct": 25.0, + "pass_pct": 23.529411, "task_pass_equiv": 2.05, "task_pass_pct": 20.5 } }, "totals": { "tasks": 22, - "total_tests": 59, + "total_tests": 61, "passed_tests": 13, - "pass_pct": 22.033897, + "pass_pct": 21.311476, "task_pass_equiv": 3.3833334, "task_pass_pct": 15.378788 } @@ -90,19 +90,19 @@ "schema": { "tasks": 10, "total_tests": 34, - "passed_tests": 23, - "pass_pct": 67.64706, - "task_pass_equiv": 6.5333333, - "task_pass_pct": 65.33333 + "passed_tests": 22, + "pass_pct": 64.70588, + "task_pass_equiv": 6.2, + "task_pass_pct": 62.0 } }, "totals": { "tasks": 22, "total_tests": 61, - "passed_tests": 49, - "pass_pct": 80.327866, - "task_pass_equiv": 17.533335, - "task_pass_pct": 79.696976 + "passed_tests": 47, + "pass_pct": 77.04918, + "task_pass_equiv": 16.2, + "task_pass_pct": 73.63637 } } }