Architecture Name
Dynamic Memory Table
Parent issue
#14
Motivations
Memory tables (DeepSeek Engram, LongCat, L3, etc.) can expand model capacity while efficiently reading only a small subset of parameters per token. This provides a promising way to increase stored knowledge without proportionally increasing active computation.
Existing approaches often determine candidate memories from tokens or local token patterns. Although context may affect how retrieved information is used, the accessible memory itself remains relatively static.
DMT asks whether memory access can adapt more directly to the model’s current semantic state while preserving fast, sparse lookup. We hypothesize that this could improve not only knowledge recall, but also tasks requiring contextualization and composition of retrieved information.
Proposed Architecture
DMT adds an internal parametric memory table to OLMo 3 with three properties:
- scalable memory capacity;
- context-dependent memory reading;
- bounded sparse access per token.
Preliminary Results (if any)
Related work supports memory tables as a promising scaling direction:
Experiments Plan
Compare OLMo 3, a capacity-matched static memory table, DMT, and an active-compute-matched control using the same data and token budget.
Evaluate language-modeling loss, knowledge recall, compositional reasoning, and long-context performance. Ablations will vary memory capacity, access budget, and static versus dynamic reading.
Initial multi-seed experiments will use smaller OLMo 3-compatible models before larger-scale validation. Success requires DMT to consistently outperform both OLMo 3 and the static-memory control, including on tasks beyond local recall, while keeping per-token memory access bounded.
Architecture Name
Dynamic Memory Table
Parent issue
#14
Motivations
Memory tables (DeepSeek Engram, LongCat, L3, etc.) can expand model capacity while efficiently reading only a small subset of parameters per token. This provides a promising way to increase stored knowledge without proportionally increasing active computation.
Existing approaches often determine candidate memories from tokens or local token patterns. Although context may affect how retrieved information is used, the accessible memory itself remains relatively static.
DMT asks whether memory access can adapt more directly to the model’s current semantic state while preserving fast, sparse lookup. We hypothesize that this could improve not only knowledge recall, but also tasks requiring contextualization and composition of retrieved information.
Proposed Architecture
DMT adds an internal parametric memory table to OLMo 3 with three properties:
Preliminary Results (if any)
Related work supports memory tables as a promising scaling direction:
Experiments Plan
Compare OLMo 3, a capacity-matched static memory table, DMT, and an active-compute-matched control using the same data and token budget.
Evaluate language-modeling loss, knowledge recall, compositional reasoning, and long-context performance. Ablations will vary memory capacity, access budget, and static versus dynamic reading.
Initial multi-seed experiments will use smaller OLMo 3-compatible models before larger-scale validation. Success requires DMT to consistently outperform both OLMo 3 and the static-memory control, including on tasks beyond local recall, while keeping per-token memory access bounded.