mlsanityData to 7 Oct 2026

Memory Layers at Scale

arXiv 2412.09764 ↗First posted 12 Dec 2024cs.CL cs.AIcited by 39

Not on the map. Why?

Abstract (arXiv)

Memory layers use a trainable key-value lookup mechanism to add extra parameters to a model without increasing FLOPs. Conceptually, sparsely activated memory layers complement compute-heavy dense feed-forward layers, providing dedicated capacity to store and retrieve information cheaply. This work takes memory layers beyond proof-of-concept, proving their utility at contemporary scale. On downstream tasks, language models augmented with our improved memory layer outperform dense models with more than twice the computation budget, as well as mixture-of-expert models when matched for both compute and parameters. We find gains are especially pronounced for factual tasks. We provide a fully parallelizable memory layer implementation, demonstrating scaling laws with up to 128B memory parameters, pretrained to 1 trillion tokens, comparing to base models with up to 8B parameters.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 30 of its 36 references that are in the map. First posted on arXiv 12 Dec 2024.

Builds on

30 of 36 references.

References · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  9. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  10. 2021Program Synthesis with Large Language Modelscited by 3,509
  11. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  12. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
18 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  2. 2022Emergent Abilities of Large Language Modelscited by 2,483
  3. 2017Billion-scale similarity search with GPUscited by 2,477
  4. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  5. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  6. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  8. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  9. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  10. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  11. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  12. 2014Neural Turing Machinescited by 630
  13. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  14. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  15. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  16. 2022Predictability and Surprise in Large Generative Modelscited by 209
  17. 2019Large Memory Layers with Product Keyscited by 120
  18. 2024Mixture of A Million Expertscited by 70

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 15; 2026: 22.

39 papers.

Sort:

By area:

  1. 2024Titans: Learning to Memorize at Test Timecited by 303
  2. 2026Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Modelscited by 78
  3. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routingcited by 53
  4. 2025Mem-α: Learning Memory Construction via Reinforcement Learningcited by 49
  5. 2025Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topicscited by 36
  6. 2025Position: Episodic Memory is the Missing Piece for Long-Term LLM Agentscited by 35
  7. 2025SonicMoE: Accelerating MoE with IO and Tile-aware Optimizationscited by 20
  8. 2025Scaling Embedding Layers in Language Modelscited by 19
  9. 2025From Bytes to Ideas: Language Modeling with Autoregressive U-Netscited by 15
  10. 2025Pretraining with hierarchical memories: separating long-tail and common knowledgecited by 13
  11. 2025General Intelligence Requires Reward-based Pretrainingcited by 12
  12. 2025Learning Facts at Scale with Active Readingcited by 11
27 more
  1. 2026Fast-weight Product Key Memorycited by 11
  2. 2025Memory-Augmented Transformers: A Systematic Review from Neuroscience Principles to Enhanced Model Architecturescited by 9
  3. 2025UltraMemV2: Memory Networks Scaling to 120B Parameters with Superior Long-Context Learningcited by 5
  4. 2025Goal-Directed Search Outperforms Goal-Agnostic Memory Compression in Long-Context Memory Taskscited by 4
  5. 2026Frozen Memory Is Not Enough: Rethinking External Memory as Extractioncited by 3
  6. 2025On Memory: A comparison of memory mechanisms in world modelscited by 2
  7. 2026JTok: On Token Embedding as another Axis of Scaling Law via Joint Token Self-modulationcited by 2
  8. 2026User as Engram: Internalizing Per-User Memory as Local Parametric Editscited by 2
  9. 2026MemSFT: Mitigating Alignment Tax with an External Parametric Memorycited by 2
  10. 2025COSMIR: Chain Orchestrated Structured Memory for Iterative Reasoning over Long Contextcited by 1
  11. 2026Improving Sparse Memory Finetuningcited by 1
  12. 2026AdMem: Advanced Memory for Task-solving Agentscited by 1
  13. 2026MARCH: Scaling Recurrent Memory with Content-Routed State Anchorscited by 1
  14. 2026Programs-of-Layers in LLMs through the Lens of Cortical Areascited by 1
  15. 2025Mixture of Lookup Key-Value Expertscited by 0
  16. 2026Mixture of Chapters: Scaling Learnt Memory in Transformerscited by 0
  17. 2026Sparse Memory Finetuning as a Low-Forgetting Alternative to LoRA and Full Finetuningcited by 0
  18. 2026Repeated Shared Access Enables Grokking, but Edit Propagation Depends on an Addressable Memorycited by 0
  19. 2026Co-LMLM: Continuous-Query Limited Memory Language Modelscited by 0
  20. 2026More Than Memory: Task-Conditioned Signed FFN Writes in Long-Context Retrievalcited by 0
  21. 2026Kathleen Remembers: Length-Invariant One-Shot Recall Without Attentioncited by 0
  22. 2026Transfiver: Human-AI Co-Inference through a Shared Editable Statecited by 0
  23. 2026MoME: Mixture-of-Memory Embeddings for Context-Aware Sparse Lookupcited by 0
  24. 2026Infinite-Parameter LLMs: Generating and Adapting Weights from Live Datacited by 0
  25. 2026Single-Layer MeMo as a Randomized Hamming-Kernel Classifiercited by 0
  26. 2026MoRE: Scaling mixture of experts with hardware-aware low-rank routingcited by 0
  27. 2026Forking: Sudden Overfitting Under Replaycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory
  2. 2024Layer-Condensed KV Cache for Efficient Inference of Large Language Models
  3. 2024Accelerating Large Language Model Inference with Self-Supervised Early Exits
  4. 2025SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deployment
  5. 2026A Single-Layer Model Can Do Language Modeling
  6. 2026Understand and Accelerate Memory Processing Pipeline for Large Language Model Inference
  7. 2025Accelerating Large Language Model Inference via Early-Exiting Algorithms
  8. 2026Universal Test-Time Training