mlsanityData to 7 Oct 2026

Improving language models by retrieving from trillions of tokens

arXiv 2112.04426 ↗First posted 8 Dec 2021cs.CL cs.LGcited by 1,278

Not on the map. Why?

Cited by 423 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We enhance auto-regressive language models by conditioning on document chunks retrieved from a large corpus, based on local similarity with preceding tokens. With a 2 trillion token database, our Retrieval-Enhanced Transformer (RETRO) obtains comparable performance to GPT-3 and Jurassic-1 on the Pile, despite using 25× fewer parameters. After fine-tuning, RETRO performance translates to downstream knowledge-intensive tasks such as question answering. RETRO combines a frozen Bert retriever, a differentiable encoder and a chunked cross-attention mechanism to predict tokens based on an order of magnitude more data than what is typically consumed during training. We typically train RETRO from scratch, yet can also rapidly RETROfit pre-trained transformers with retrieval and still achieve good performance. Our work opens up new avenues for improving language models through explicit memory at unprecedented scale.

Key facts. Cited by 1,278 papers in the map (data through 7 Oct 2026). Builds on 41 of its 64 references that are in the map. First posted on arXiv 8 Dec 2021.

Builds on

41 of 64 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2016Deep Learning with Differential Privacycited by 2,682
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  11. 2020Extracting Training Data from Large Language Modelscited by 2,144
  12. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
29 more
  1. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  2. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  3. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  4. 2019Root Mean Square Layer Normalizationcited by 1,425
  5. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  6. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  7. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  8. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  9. 2019Attention is not Explanationcited by 916
  10. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  11. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  12. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  13. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  14. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  15. 2019Green AIcited by 620
  16. 2013Generating Sequences With Recurrent Neural Networkscited by 549
  17. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  18. 2021Internet-Augmented Dialogue Generationcited by 217
  19. 2016Exploring the Limits of Language Modelingcited by 172
  20. 2020Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasetscited by 137
  21. 2021End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answeringcited by 133
  22. 2019Lessons from Archives: Strategies for Collecting Sociocultural Data in Machine Learningcited by 116
  23. 2021Adaptive Semiparametric Language Modelscited by 83
  24. 2016Improving Neural Language Models with a Continuous Cachecited by 81
  25. 2018Guiding Neural Machine Translation with Retrieved Translation Piecescited by 63
  26. 2020A Memory Efficient Baseline for Open Domain Question Answeringcited by 32
  27. 2020Guided Transformer: Leveraging Multiple External Sources for Representation Learning in Conversational Searchcited by 23
  28. 2016A Neural Knowledge Language Modelcited by 13
  29. 2019WikiLinkGraphs: A Complete, Longitudinal and Multi-Language Dataset of the Wikipedia Link Networkscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 118; 2023: 256; 2024: 324; 2025: 296; 2026: 282.

1,278 papers.

Sort:

By area:

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2023A Survey of Large Language Modelscited by 2,537
  7. 2022Emergent Abilities of Large Language Modelscited by 2,483
  8. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  9. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  10. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  11. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  12. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
38 more
  1. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  2. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  3. 2022Diffusion Models in Vision: A Surveycited by 1,008
  4. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,000
  5. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  6. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  7. 2022A Generalist Agentcited by 802
  8. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  9. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  10. 2022Galactica: A Large Language Model for Sciencecited by 736
  11. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  12. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  13. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  14. 2023A Comprehensive Overview of Large Language Modelscited by 639
  15. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  16. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  17. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 556
  18. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  19. 2024RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrievalcited by 519
  20. 2023Active Retrieval Augmented Generationcited by 506
  21. 2024Large Language Models: A Surveycited by 506
  22. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  23. 2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexitycited by 437
  24. 2023Cognitive Architectures for Language Agentscited by 422
  25. 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 414
  26. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 408
  27. 2023Augmented Language Models: a Surveycited by 398
  28. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 367
  29. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  30. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  31. 2023Adapting Language Models to Compress Contextscited by 350
  32. 2025Agentic Context Engineering: Evolving Contexts for Self-Improving Language Modelscited by 349
  33. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
  34. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  35. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 308
  36. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 307
  37. 2023Tool Learning with Foundation Modelscited by 302
  38. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302

50 most cited of 1,278 All 1,278 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025MLP Memory: A Retriever-Pretrained Memory for Large Language Models
  2. 2026DreamingGoose: Staged Distillation from Autoregressive Transformers to Bidirectional Recurrent Diffusion Language Models
  3. 2024Embedding-Informed Adaptive Retrieval-Augmented Generation of Large Language Models
  4. 2023Improving Language Models via Plug-and-Play Retrieval Feedback
  5. 2026Distilling Sequential Computation in Transformer Language Models
  6. 2021RETRONLU: Retrieval Augmented Task-Oriented Semantic Parsing
  7. 2026ReFilter: Improving Robustness of Retrieval-Augmented Generation via Gated Filter
  8. 2026A Family of LLMs Liberated from Static Vocabularies