mlsanityData to 7 Oct 2026

Large Memory Layers with Product Keys

arXiv 1907.05242 ↗First posted 10 Jul 2019cs.CL cs.LGcited by 120

Not on the map. Why?

Cited by 25 in Mixture-of-experts, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper introduces a structured memory which can be easily integrated into a neural network. The memory is very large by design and significantly increases the capacity of the architecture, by up to a billion parameters with a negligible computational overhead. Its design and access pattern is based on product keys, which enable fast and exact nearest neighbor search. The ability to increase the number of parameters while keeping the same computational budget lets the overall system strike a better trade-off between prediction accuracy and computation efficiency both at training and test time. This memory layer allows us to tackle very large scale language modeling tasks. In our experiments we consider a dataset with up to 30 billion words, and we plug our memory layer in a state-of-the-art transformer-based architecture. In particular, we found that a memory augmented model with only 12 layers outperforms a baseline transformer model with 24 layers, while being twice faster at inference time. We release our code for reproducibility purposes.

Key facts. Cited by 120 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Jul 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 3; 2021: 18; 2022: 11; 2023: 14; 2024: 14; 2025: 16; 2026: 44.

120 papers.

Sort:

By area:

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  3. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  4. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  5. 2021A Survey of Transformerscited by 399
  6. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  7. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  8. 2021Hash Layers For Large Sparse Modelscited by 236
  9. 2022Memorizing Transformerscited by 219
  10. 2022Recurrent Memory Transformercited by 197
  11. 2021The Efficiency Misnomercited by 98
  12. 2024CT2Rep: Automated Radiology Report Generation for 3D Medical Imagingcited by 96
38 more
  1. 2023Fast Inference of Mixture-of-Experts Language Models with Offloadingcited by 88
  2. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
  3. 2026Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Modelscited by 78
  4. 2024Mixture of A Million Expertscited by 70
  5. 2021ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Informationcited by 65
  6. 2023SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficientcited by 58
  7. 2021NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Searchcited by 55
  8. 2023A Study on ReLU and Softmax in Transformercited by 53
  9. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routingcited by 53
  10. 2020Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Expertscited by 43
  11. 2022Distinguishing Homophenes Using Multi-Head Visual-Audio Memory for Lip Readingcited by 39
  12. 2024Memory Layers at Scalecited by 39
  13. 2022An Efficient Memory-Augmented Transformer for Knowledge-Intensive NLP Taskscited by 36
  14. 2021Efficient Transformers in Reinforcement Learning using Actor-Learner Distillationcited by 35
  15. 2021Not All Memories are Created Equal: Learning to Forget by Expiringcited by 32
  16. 2023Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizercited by 28
  17. 2023Approximating Two-Layer Feedforward Networks for Efficient Transformerscited by 27
  18. 2024XMoE: Sparse Models with Fine-grained and Adaptive Expert Selectioncited by 27
  19. 2023Transformer-VQ: Linear-Time Transformers via Vector Quantizationcited by 26
  20. 2021Pruning Self-attentions into Convolutional Layers in Single Pathcited by 24
  21. 2025Mixture of Experts Made Intrinsically Interpretablecited by 22
  22. 2022NTIRE 2022 Challenge on Super-Resolution and Quality Enhancement of Compressed Video: Dataset, Methods and Resultscited by 20
  23. 2024Ultra-Sparse Memory Networkcited by 19
  24. 2025Scaling Embedding Layers in Language Modelscited by 19
  25. 2022Discrete Key-Value Bottleneckcited by 18
  26. 2025Continual Learning via Sparse Memory Finetuningcited by 17
  27. 2021NWT: Towards natural audio-to-video generation with representation learningcited by 14
  28. 2022Turbo Training with Token Dropoutcited by 14
  29. 2022N-Grammer: Augmenting Transformers with latent n-gramscited by 13
  30. 2023Alternating Updates for Efficient Transformerscited by 13
  31. 2023ResMem: Learn what you can and memorize the restcited by 13
  32. 2025Pretraining with hierarchical memories: separating long-tail and common knowledgecited by 13
  33. 2022Sparse Mixers: Combining MoE and Mixing to build a more efficient BERTcited by 12
  34. 2024Monet: Mixture of Monosemantic Experts for Transformerscited by 12
  35. 2024Retrieval-Enhanced Machine Learning: Synthesis and Opportunitiescited by 11
  36. 2026Fast-weight Product Key Memorycited by 11
  37. 2026Adaptive Loops and Memory in Transformers: Think Harder or Know More?cited by 11
  38. 2026STEM: Scaling Transformers with Embedding Modulescited by 10

50 most cited of 120 All 120 (JSON)