Architecture Name
AHN
Parent issue
#5
Motivations
Long-context modeling faces a fundamental trade-off between the fidelity of full attention and the efficiency of bounded-memory sequence models. Causal attention preserves exact historical information through a growing KV cache, but its cache grows linearly with context length and its attention cost grows quadratically. RNN-like models instead maintain a fixed-size state with constant per-token update cost, but compress all history and can lose information required for precise long-range recall.
Artificial Hippocampus Networks proposes a hybrid memory mechanism inspired by the multi-store model of memory. A sliding attention window retains recent KV pairs as lossless short-term memory. When a KV pair leaves the window, a learnable recurrent module compresses it into a fixed-size long-term memory state. The current query reads from both the in-window KV cache and the compressed state.
This proposal evaluates whether AHN can retain the long-context quality of full attention more effectively than sliding-window attention under the same lossless-memory budget, while reducing KV-cache growth and long-sequence compute. It also tests whether the architecture remains compatible with the full ArchSpace model lifecycle and preserves short-context capability.
Proposed Architecture
AHN augments each causal attention layer with a recurrent memory pathway. Let $(k_{t-W}, v_{t-W})$ be the KV pair that just exits a sliding attention window of size $W$. AHN updates its fixed-size memory state as
$$
h_{t-W}=\mathrm{AHN}((k_{t-W}, v_{t-W}), h_{t-W-1}).
$$
The token at position $t$ retains the lossless in-window memory
$$
{(k_i,v_i)}_{i=t-W+1}^{t},
$$
and reads from both this window and the compressed history:
$$
y_t=f\left(h_{t-W}, {(k_i,v_i)}_{i=t-W+1}^{t}, q_t\right).
$$
For the initial implementation, we propose AHN-GatedDeltaNet (AHN-GDN), the paper's representative instantiation. Per attention head, AHN-GDN consumes the KV pair leaving the window and updates the recurrent state with a gated delta rule. The current query then reads from the compressed state through a gated, query-dependent projection, and this output is added to the sliding-window attention output.
For inputs with length no greater than $W$, AHN remains inactive and the model follows the parent Transformer path. For longer inputs, AHN replaces unbounded historical KV storage with a fixed-size compressed state while preserving standard autoregressive token-by-token decoding.
The initial proposal is limited to AHN-GDN. AHN-Mamba2 and AHN-DeltaNet should be considered only after the AHN-GDN implementation is validated.
Preliminary Results
Experiments Plan
All comparisons will use matched parent architecture, data, token budget, context length, initialization, learning-rate schedule, and evaluation intervals. Full-attention, sliding-window, and AHN variants will use the same lossless-memory budget where applicable.
Architecture Name
AHN
Parent issue
#5
Motivations
Long-context modeling faces a fundamental trade-off between the fidelity of full attention and the efficiency of bounded-memory sequence models. Causal attention preserves exact historical information through a growing KV cache, but its cache grows linearly with context length and its attention cost grows quadratically. RNN-like models instead maintain a fixed-size state with constant per-token update cost, but compress all history and can lose information required for precise long-range recall.
Artificial Hippocampus Networks proposes a hybrid memory mechanism inspired by the multi-store model of memory. A sliding attention window retains recent KV pairs as lossless short-term memory. When a KV pair leaves the window, a learnable recurrent module compresses it into a fixed-size long-term memory state. The current query reads from both the in-window KV cache and the compressed state.
This proposal evaluates whether AHN can retain the long-context quality of full attention more effectively than sliding-window attention under the same lossless-memory budget, while reducing KV-cache growth and long-sequence compute. It also tests whether the architecture remains compatible with the full ArchSpace model lifecycle and preserves short-context capability.
Proposed Architecture
AHN augments each causal attention layer with a recurrent memory pathway. Let$(k_{t-W}, v_{t-W})$ be the KV pair that just exits a sliding attention window of size $W$ . AHN updates its fixed-size memory state as
The token at position$t$ retains the lossless in-window memory
and reads from both this window and the compressed history:
For the initial implementation, we propose AHN-GatedDeltaNet (AHN-GDN), the paper's representative instantiation. Per attention head, AHN-GDN consumes the KV pair leaving the window and updates the recurrent state with a gated delta rule. The current query then reads from the compressed state through a gated, query-dependent projection, and this output is added to the sliding-window attention output.
For inputs with length no greater than$W$ , AHN remains inactive and the model follows the parent Transformer path. For longer inputs, AHN replaces unbounded historical KV storage with a fixed-size compressed state while preserving standard autoregressive token-by-token decoding.
The initial proposal is limited to AHN-GDN. AHN-Mamba2 and AHN-DeltaNet should be considered only after the AHN-GDN implementation is validated.
Preliminary Results
Experiments Plan
All comparisons will use matched parent architecture, data, token budget, context length, initialization, learning-rate schedule, and evaluation intervals. Full-attention, sliding-window, and AHN variants will use the same lossless-memory budget where applicable.