mlsanityData to 7 Oct 2026

Improving Neural Language Models with a Continuous Cache

arXiv 1612.04426 ↗First posted 13 Dec 2016cs.CL cs.LGcited by 81

Not on the map. Why?

Abstract (arXiv)

We propose an extension to neural network language models to adapt their prediction to the recent history. Our model is a simplified version of memory augmented networks, which stores past hidden activations as memory and accesses them through a dot product with the current hidden activation. This mechanism is very efficient and scales to very large memory sizes. We also draw a link between the use of external memory in neural network and cache models used with count based language models. We demonstrate on several language model datasets that our approach performs significantly better than recent memory augmented networks.

Key facts. Cited by 81 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Dec 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 21; 2022: 24; 2023: 16; 2024: 12; 2025: 3; 2026: 5.

81 papers.

Sort:

By area:

  1. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  2. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  3. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  4. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  5. 2023Augmented Language Models: a Surveycited by 398
  6. 2023Think before you speak: Training Language Models With Pause Tokenscited by 254
  7. 2022Memorizing Transformerscited by 219
  8. 2021Internet-Augmented Dialogue Generationcited by 217
  9. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  10. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 130
  11. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  12. 2022Training Language Models with Memory Augmentationcited by 112
38 more
  1. 2023Lift Yourself Up: Retrieval-augmented Text Generation with Self Memorycited by 103
  2. 2022Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completioncited by 102
  3. 2021Adaptive Semiparametric Language Modelscited by 83
  4. 2022A Survey of Knowledge Enhanced Pre-trained Language Modelscited by 76
  5. 2022Your Transformer May Not be as Powerful as You Expectcited by 55
  6. 2021On the Evaluation of Neural Code Summarizationcited by 47
  7. 2021Paradigm Shift in Natural Language Processingcited by 39
  8. 2024Recent Advances in Generative AI and Large Language Models: Current Status, Challenges, and Perspectivescited by 30
  9. 2021Selfish Sparse RNN Trainingcited by 26
  10. 2022Efficient recurrent architectures through activity sparsity and sparse back-propagation through timecited by 26
  11. 2024RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processingcited by 25
  12. 2024ALR2: A Retrieve-then-Reason Framework for Long-context Question Answeringcited by 25
  13. 2022Relational Memory Augmented Language Modelscited by 24
  14. 2022Knowledge-in-Context: Towards Knowledgeable Semi-Parametric Language Modelscited by 24
  15. 2024Fuzz-Testing Meets LLM-Based Agents: An Automated and Efficient Framework for Jailbreaking Text-To-Image Generation Modelscited by 24
  16. 2025Pre-training under infinite computecited by 24
  17. 2022You can't pick your neighbors, or can you? When and how to rely on retrieval in the kNN-LMcited by 18
  18. 2024KBLaM: Knowledge Base augmented Language Modelcited by 16
  19. 2024Frozen Feature Augmentation for Few-Shot Image Classificationcited by 13
  20. 2023Recurrent Attention Networks for Long-text Modelingcited by 12
  21. 2021Beyond Short Clips: End-to-End Video-Level Learning with Collaborative Memoriescited by 11
  22. 2024Retrieval-Enhanced Machine Learning: Synthesis and Opportunitiescited by 11
  23. 2021Low-Resource Language Modelling of South African Languagescited by 10
  24. 2023KNN-LM Does Not Improve Open-ended Text Generationcited by 10
  25. 2024Efficient Prompt Caching via Embedding Similaritycited by 9
  26. 2021Capturing Structural Locality in Non-parametric Language Modelscited by 8
  27. 2021Faster Nearest Neighbor Machine Translationcited by 8
  28. 2022Zemi: Learning Zero-Shot Semi-Parametric Language Models from Multiple Taskscited by 8
  29. 2021∞-former: Infinite Memory Transformercited by 7
  30. 2024Vocabulary-Defined Semantics: Latent Space Clustering for Improving In-Context Learningcited by 7
  31. 2022Semi-Parametric Inducing Point Networks and Neural Processescited by 6
  32. 2022A Natural Bias for Language Generation Modelscited by 6
  33. 2023HistAlign: Improving Context Dependency in Language Generation by Aligning with Historycited by 6
  34. 2022Better Language Model with Hypernym Class Predictioncited by 5
  35. 2022Embedding Recycling for Language Modelscited by 5
  36. 2023TAPIR: Learning Adaptive Revision for Incremental Natural Language Understanding with a Two-Pass Modelcited by 5
  37. 2023Approximate Caching for Efficiently Serving Diffusion Modelscited by 5
  38. 2024NNOSE: Nearest Neighbor Occupational Skill Extractioncited by 5

50 most cited of 81 All 81 (JSON)