MoNe: Modular Neural Memory for Efficient Long Context Inference
Not on the map. Why?
Abstract (arXiv)
We present MoNe, a lightweight modular neural memory that attaches to any frozen pretrained Transformer to enable long-context inference without retraining. MoNe reads context in fixed-size segments via test-time learning of fast-weight neural memory networks with layer-localized gradient updates; at inference, the memory generates keys and values from the query tokens alone, with no context tokens re-read. This two-phase design decouples inference cost from context length, achieving O(N) preprocessing and O(1) query cost with peak GPU memory that does not grow with N. At 128K tokens, MoNe reduces both compute and peak GPU memory by approximately 80% compared to ICL with only 6.4% parameter overhead. MoNe generalizes to context lengths far beyond the backbone's native window, achieving strong performance on needle-in-a-haystack and word extraction benchmarks from RULER, where ICL degrades sharply.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 32 of its 36 references that are in the map. First posted on arXiv 18 Aug 2026.
Builds on
32 of 36 references.
References · 32
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
- 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
- 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
- 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
- 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
- 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
- 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
- 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
- 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 800
20 more
- 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memorycited by 722
- 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
- 2023LaMP: When Large Language Models Meet Personalizationcited by 425
- 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
- 2024Titans: Learning to Memorize at Test Timecited by 303
- 2025MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agentscited by 226
- 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
- 2024How to Train Long-Context Language Models (Effectively)cited by 141
- 2025Test-Time Training Done Rightcited by 121
- 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 119
- 2023Pearl: Personalizing Large Language Model Writing Assistants with Generation-Calibrated Retrieverscited by 91
- 2024Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queriescited by 85
- 2024RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculturecited by 80
- 2025PENCIL: Long Thoughts with Short Memorycited by 31
- 2026MEMENTO: Teaching LLMs to Manage Their Own Contextcited by 16
- 2023Complementary Domain Adaptation and Generalization for Unsupervised Continual Domain Shift Learningcited by 9
- 2026Position: Modular Memory is the Key to Continual Learning Agentscited by 7
- 2025PiCa: Parameter-Efficient Fine-Tuning with Column Space Projectioncited by 5
- 2024Hollowed Net for On-Device Personalization of Text-to-Image Diffusion Modelscited by 4
- 2026Memory-Efficient Fine-Tuning Diffusion Transformers via Dynamic Patch Sampling and Block Skippingcited by 1
Built on by
Not cited yet.