mlsanityData to 7 Oct 2026

Training Language Models with Memory Augmentation

arXiv 2205.12674 ↗First posted 25 May 2022cs.CL cs.LGcited by 112

Not on the map. Why?

Cited by 29 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent work has improved language models (LMs) remarkably by equipping them with a non-parametric memory component. However, most existing approaches only introduce mem-ories at testing time or represent them using a separately trained encoder, resulting in suboptimal training of the language model. In this work, we present TRIME, a novel yet simple training approach designed for training LMs with memory augmentation. Our approach uses a training objective that directly takes in-batch examples as accessible memory. We also present new methods for memory construction and data batching, which are used for adapting to different sets of memories--local, long-term, and external memory--at testing time. We evaluate TRIME on multiple language modeling and machine translation benchmarks and show that it is able to achieve significant improvements across all the settings. Concretely, TRIME reduces the perplexity from 18.70 to 15.37 on WIKITEXT-103, by effectively leveraging a large memory set from the training corpus. Compared to standard LM training, TRIME adds negligible computational overhead and is compatible with different neural architectures, making it a versatile solution for training memory-augmented LMs.

Key facts. Cited by 112 papers in the map (data through 7 Oct 2026). Builds on 49 of its 61 references that are in the map. First posted on arXiv 25 May 2022.

Builds on

49 of 61 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2020Longformer: The Long-Document Transformercited by 3,362
  9. 2016Pointer Sentinel Mixture Modelscited by 2,992
  10. 2017Billion-scale similarity search with GPUscited by 2,477
  11. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  12. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
37 more
  1. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  2. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  3. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  4. 2020Rethinking Attention with Performerscited by 1,584
  5. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  7. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  8. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  9. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  10. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  11. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  12. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  13. 2020Efficient Transformers: A Surveycited by 784
  14. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  15. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  16. 2021Random Feature Attentioncited by 350
  17. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  18. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  19. 2022Memorizing Transformerscited by 219
  20. 2020Nearest Neighbor Machine Translationcited by 209
  21. 2019Episodic Memory in Lifelong Language Learningcited by 188
  22. 2019Adaptive Attention Span in Transformerscited by 148
  23. 2018Character-Level Language Modeling with Deeper Self-Attentioncited by 138
  24. 2021End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answeringcited by 133
  25. 2015Ask Me Anything: Dynamic Memory Networks for Natural Language Processingcited by 116
  26. 2021Efficient Nearest Neighbor Language Modelscited by 93
  27. 2021Adaptive Semiparametric Language Modelscited by 83
  28. 2016Improving Neural Language Models with a Continuous Cachecited by 81
  29. 2021Adaptive Nearest Neighbor Machine Translationcited by 58
  30. 2020Augmenting Transformers with KNN-Based Composite Memory for Dialoguecited by 53
  31. 2021Mention Memory: incorporating textual knowledge into Transformers through entity mention attentioncited by 42
  32. 2017Unbounded cache model for online language modeling with open vocabularycited by 37
  33. 2021When Attention Meets Fast Recurrence: Training Language Models with Reduced Computecited by 34
  34. 2021Not All Memories are Created Equal: Learning to Forget by Expiringcited by 32
  35. 2021GNN-LM: Language Modeling based on Global Contexts via GNNcited by 27
  36. 2021ReadTwice: Reading Very Large Documents with Memoriescited by 14
  37. 2022LaMemo: Language Modeling with Look-Ahead Memorycited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 11; 2023: 47; 2024: 40; 2025: 8; 2026: 6.

112 papers.

Sort:

By area:

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  2. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  3. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  4. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 656
  5. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  6. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  7. 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 414
  8. 2023Augmented Language Models: a Surveycited by 398
  9. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  10. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
  11. 2023Adapting Language Models to Compress Contextscited by 350
  12. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
38 more
  1. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  2. 2023RA-DIT: Retrieval-Augmented Dual Instruction Tuningcited by 194
  3. 2023Augmenting Language Models with Long-Term Memorycited by 186
  4. 2023Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Modelscited by 180
  5. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  6. 2023LM vs LM: Detecting Factual Errors via Cross Examinationcited by 155
  7. 2024Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokenscited by 148
  8. 2025Dynamic Cheatsheet: Test-Time Learning with Adaptive Memorycited by 146
  9. 2023ChatDB: Augmenting LLMs with Databases as Their Symbolic Memorycited by 136
  10. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  11. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 127
  12. 2024xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Tokencited by 123
  13. 2023Complex Claim Verification with Evidence Retrieved in the Wildcited by 110
  14. 2023Lift Yourself Up: Retrieval-augmented Text Generation with Self Memorycited by 103
  15. 2022Training Trajectories of Language Models Across Scalescited by 80
  16. 2024Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalizationcited by 78
  17. 2023In-context Pretraining: Language Modeling Beyond Document Boundariescited by 76
  18. 2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
  19. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  20. 2024Retrieval-Augmented Generation for Natural Language Processing: A Surveycited by 70
  21. 2023RET-LLM: Towards a General Read-Write Memory for Large Language Modelscited by 64
  22. 2026Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontierscited by 60
  23. 2024DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent)cited by 56
  24. 2024Context-DPO: Aligning Language Models for Context-Faithfulnesscited by 50
  25. 2024Enhancing Contextual Understanding in Large Language Models through Contrastive Decodingcited by 49
  26. 2023Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Modelscited by 46
  27. 2022Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformercited by 45
  28. 2023KCTS: Knowledge-Constrained Tree Search Decoding with Token-Level Hallucination Detectioncited by 43
  29. 2024Training Language Models to Generate Text with Citations via Fine-grained Rewardscited by 40
  30. 2022Nonparametric Masked Language Modelingcited by 39
  31. 2024Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-trainingcited by 37
  32. 2022Can Retriever-Augmented Language Models Reason? The Blame Game Between the Retriever and the Language Modelcited by 32
  33. 2024The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Surveycited by 32
  34. 2023Privacy Implications of Retrieval-Based Language Modelscited by 31
  35. 2023Why do Nearest Neighbor Language Models Work?cited by 29
  36. 2023Emptying the Ocean with a Spoon: Should We Edit Models?cited by 27
  37. 2024Metacognitive Retrieval-Augmented Large Language Modelscited by 26
  38. 2024RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processingcited by 25

50 most cited of 112 All 112 (JSON)