mlsanityData to 7 Oct 2026

More Room for Language: Investigating the Effect of Retrieval on Language Models

arXiv 2404.10939 ↗First posted 16 Apr 2024cs.CLcited by 2

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

Retrieval-augmented language models pose a promising alternative to standard language modeling. During pretraining, these models search in a corpus of documents for contextually relevant information that could aid the language modeling objective. We introduce an 'ideal retrieval' methodology to study these models in a fully controllable setting. We conduct an extensive evaluation to examine how retrieval augmentation affects the behavior of the underlying language model. Among other things, we observe that these models: i) save substantially less world knowledge in their weights, ii) are better at understanding local context and inter-word dependencies, but iii) are worse at comprehending global context.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 35 of its 66 references that are in the map. First posted on arXiv 16 Apr 2024.

Builds on

35 of 66 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Surface-Based Retrieval Reduces Perplexity of Retrieval-Augmented Language Models25 May
  2. 2023Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study13 Apr
  3. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Models5 Aug
  4. 2020REALM: Retrieval-Augmented Language Model Pre-Training10 Feb

Other references · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  9. 2019Language Models as Knowledge Bases?cited by 1,922
  10. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  11. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  12. 2016Understanding intermediate layers using linear classifier probescited by 1,279
19 more
  1. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  2. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  3. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  4. 2018Neural Network Acceptability Judgmentscited by 952
  5. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  6. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  7. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  8. 2021Editing Factual Knowledge in Language Modelscited by 548
  9. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  10. 2016Deep Biaffine Attention for Neural Dependency Parsingcited by 334
  11. 2019BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Modelcited by 184
  12. 201975 Languages, 1 Model: Parsing Universal Dependencies Universallycited by 90
  13. 2019Do Attention Heads in BERT Track Syntactic Dependencies?cited by 87
  14. 2023Trained on 100 million words and still in shape: BERT meets British National Corpuscited by 39
  15. 2017An optimal transportation approach for assessing almost stochastic ordercited by 33
  16. 2021Attention Can Reflect Syntactic Structure (If You Let It)cited by 22
  17. 2021Boosted Dense Retrievercited by 10
  18. 2023On the Generalization Ability of Retrieval-Enhanced Transformerscited by 4
  19. 2023BRENT: Bidirectional Retrieval Enhanced Norwegian Transformercited by 2

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024BERTs are Generative In-Context Learnerscited by 11
  2. 2025Studying the Role of Input-Neighbor Overlap in Retrieval-Augmented Language Models Training Efficiencycited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026To Memorize or to Retrieve: Scaling the Interaction Between Pretraining and Retrieval
  2. 2025Do Retrieval-Augmented Language Models Adapt to Varying User Needs?
  3. 2026Beyond the Parameters: A Technical Survey of Contextual Enrichment in Large Language Models: From In-Context Prompting to Causal Retrieval-Augmented Generation
  4. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Context
  5. 2024Reliable, Adaptable, and Attributable Language Models with Retrieval
  6. 2024Assessing "Implicit" Retrieval Robustness of Large Language Models
  7. 2023In-Context Retrieval-Augmented Language Models
  8. 2023Retrieval meets Long Context Large Language Models