Architecture Name
March
Parent issue
#2
Motivations
Gated DeltaNet compresses the entire causal prefix into a single recurrent state. Although efficient, earlier information may be weakened by decay or overwritten by later updates, and only the latest state remains accessible. MARCH preserves selected earlier states and makes them directly retrievable. We hypothesize that this improves long-context modeling while retaining Gated DeltaNet’s efficiency and short-context quality.
Proposed Architecture
MARCH retains the original Gated DeltaNet architecture and recurrence. After every (C=512) text tokens, it saves the current cumulative recurrent state as a state anchor without resetting the recurrence.
A learned summary position produces a routing key for each anchor. For every text token, a head-specific routing query scores all causally visible anchors together with a learned null option. The selected anchors are aggregated and read using the token’s original Gated DeltaNet query:
$$\mathbf{o}_{t,h}
=
\frac{\mathbf{S}_{t,h}\mathbf{q}_{t,h}}{\sqrt{d_k}}
+
\sum_{m \in \mathcal{M}(t)}
\pi_{t,h,m}
\frac{\mathbf{A}^{(m)}_{h}\mathbf{q}_{t,h}}{\sqrt{d_k}}.$$
The current-state branch always remains active, while the null option allows the model to ignore historical states.
Preliminary Results (if any)
No response
Experiments Plan
Experiments Plan
| Model size |
Baseline |
Proposed model |
| 1B |
OLMo 3 |
OLMo 3 + MARCH |
| 3B |
OLMo 3 |
OLMo 3 + MARCH |
| 7B |
OLMo 3 |
OLMo 3 + MARCH |
Within each model-size pair, both runs will use the same model dimensions, tokenizer, training data and order, token budget, sequence length, optimizer, learning-rate schedule, batch size, random seed, distributed topology, and evaluation protocol. The only architectural difference will be the addition of MARCH.
Comparisons will include training-loss and downstream evaluation results at matched token counts. Long-context performance, training throughput, peak memory, and inference costs will also be reported.
The proposal will be considered supported if MARCH shows consistent quality improvements across the 1B, 3B, and 7B comparisons without introducing training instability, while its additional computation and memory costs remain practically acceptable.
Architecture Name
March
Parent issue
#2
Motivations
Gated DeltaNet compresses the entire causal prefix into a single recurrent state. Although efficient, earlier information may be weakened by decay or overwritten by later updates, and only the latest state remains accessible. MARCH preserves selected earlier states and makes them directly retrievable. We hypothesize that this improves long-context modeling while retaining Gated DeltaNet’s efficiency and short-context quality.
Proposed Architecture
MARCH retains the original Gated DeltaNet architecture and recurrence. After every (C=512) text tokens, it saves the current cumulative recurrent state as a state anchor without resetting the recurrence.
A learned summary position produces a routing key for each anchor. For every text token, a head-specific routing query scores all causally visible anchors together with a learned null option. The selected anchors are aggregated and read using the token’s original Gated DeltaNet query:
The current-state branch always remains active, while the null option allows the model to ignore historical states.
Preliminary Results (if any)
No response
Experiments Plan
Experiments Plan
Within each model-size pair, both runs will use the same model dimensions, tokenizer, training data and order, token budget, sequence length, optimizer, learning-rate schedule, batch size, random seed, distributed topology, and evaluation protocol. The only architectural difference will be the addition of MARCH.
Comparisons will include training-loss and downstream evaluation results at matched token counts. Long-context performance, training throughput, peak memory, and inference costs will also be reported.
The proposal will be considered supported if MARCH shows consistent quality improvements across the 1B, 3B, and 7B comparisons without introducing training instability, while its additional computation and memory costs remain practically acceptable.