Kathleen Remembers: Length-Invariant One-Shot Recall Without Attention
Not on the map. Why?
Abstract (arXiv)
Recurrent, attention-free sequence models share a structural weakness: a fading state cannot perform exact recall of something seen once, far in the past. We add to the Kathleen trunk a second memory layer -- a "notebook": a fixed-key holographic (HRR) associative store with a learned local write gate, a self-gating raw read, and write-triggered forgetting -- 25K parameters that attach to the logits of any trunk. (1) Mechanism: on a controlled needle-in-haystack task the notebook reaches 80-82% one-shot recall at 4x the training length, where the bare trunk scores ~4% and a parameter-matched attention head scores 100% inside its training length and 0% beyond it. Addressing is length-invariant by construction; the untrained memory alone recalls at 90% accuracy identically at 512, 2048 and 4096 bytes. Because the store is a linear superposition, two capabilities follow from arithmetic alone: selective unlearning (one subtraction erases one fact to chance, retained facts unharmed) and per-token attribution (counterfactual erasure names the source fact of every correct byte, 100% provenance). (2) Real text: on WikiText-2 bytes the notebook improves prediction of repeated rare words by +0.15-0.27 bits/byte, the gain growing with the distance between mentions and holding zero-shot at 4x training length; write-triggered forgetting eliminates memory pollution at 8x length (first-mention cost +0.33 -> -0.004). (3) Scope and scale: a parameter-matched attention head does generalize on natural-text repetition, so the notebook's claim is exact recall at O(L); on a WikiText-103 ladder (8 to 512 MB) the zero-shot repeat gain rises monotonically. All experiments are pre-registered, seeds reported, and reproducible on a single free-tier GPU.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 17 of its 24 references that are in the map. First posted on arXiv 31 Aug 2026.
Builds on
17 of 24 references.
References · 17
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
- 2016Pointer Sentinel Mixture Modelscited by 3,096
- 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
- 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
- 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
- 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
- 2014Neural Turing Machinescited by 644
- 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
- 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
- 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
- 2024Titans: Learning to Memorize at Test Timecited by 328
5 more
- 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 242
- 2022Memorizing Transformerscited by 226
- 2024Memory Layers at Scalecited by 48
- 2026Kathleen: Oscillator-Based Byte-Level Text Classification Without Tokenization or Attentioncited by 1
- 2026Kathleen Writes: Autoregressive Generation and Data Scaling Without Attentioncited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026A Hippocampus for Linear Attention: An Exact Memory for What the Recurrent State Forgets
- 2026Anatomy of Associative Recall in Fixed-State Recurrences: A Matched-State Decomposition, an Interference Wall, and a Curriculum That Breaks It
- 2026The Impossibility Triangle of Long-Context Modeling
- 2025LoLA: Low-Rank Linear Attention With Sparse Caching
- 2026Echo: KV-Cache-Free Associative Recall with Spectral Koopman Operators
- 2026Memory by Design: Bayesian Layers for Sequence Modeling
- 2024Stuffed Mamba: Oversized States Lead to the Inability to Forget
- 2026Remembering Distinct Items, Not Tokens: A Learnable Dirichlet-Process Cache Between State-Space Models and Attention