mlsanityData to 7 Oct 2026

How Context Affects Language Models' Factual Predictions

arXiv 2005.04611 ↗First posted 10 May 2020cs.CLcited by 181

Not on the map. Why?

Cited by 25 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

When pre-trained on large unsupervised textual corpora, language models are able to store and retrieve factual knowledge to some extent, making it possible to use them directly for zero-shot cloze-style question answering. However, storing factual knowledge in a fixed number of weights of a language model clearly has limitations. Previous approaches have successfully provided access to information outside the model weights using supervised architectures that combine an information retrieval system with a machine reading component. In this paper, we go a step further and integrate information from a retrieval system with a pre-trained language model in a purely unsupervised way. We report that augmenting pre-trained language models in this way dramatically improves performance and that the resulting system, despite being unsupervised, is competitive with a supervised machine reading baseline. Furthermore, processing query and context with different segment tokens allows BERT to utilize its Next Sentence Prediction pre-trained classifier to determine whether the context is relevant or not, substantially improving BERT's zero-shot cloze-style question-answering performance and making its predictions robust to noisy contexts.

Key facts. Cited by 181 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 35; 2022: 32; 2023: 34; 2024: 55; 2025: 12; 2026: 12.

181 papers.

Sort:

By area:

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  3. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  4. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  5. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  6. 2022Mass-Editing Memory in a Transformercited by 976
  7. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  8. 2021GPT Understands, Toocited by 715
  9. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  10. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  11. 2021MetaICL: Learning to Learn In Contextcited by 450
  12. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
38 more
  1. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 364
  2. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
  3. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
  4. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 208
  5. 2021PAQ: 65 Million Probably-Asked Questions and What You Can Do With Themcited by 198
  6. 2021Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classificationcited by 183
  7. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  8. 2022Large Language Models with Controllable Working Memorycited by 169
  9. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  10. 2021Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Modelscited by 132
  11. 2022Autoregressive Search Engines: Generating Substrings as Document Identifierscited by 112
  12. 2021Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Basescited by 107
  13. 2023When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunitiescited by 89
  14. 2022Mind's Eye: Grounded Language Model Reasoning through Simulationcited by 83
  15. 2024Trustworthiness in Retrieval-Augmented Generation Systems: A Surveycited by 83
  16. 2021BERTese: Learning to Speak to BERTcited by 82
  17. 2023Boosting Language Models Reasoning with Chain-of-Knowledge Promptingcited by 77
  18. 2022Extracting Cultural Commonsense Knowledge at Scalecited by 76
  19. 2023LLMs4OL: Large Language Models for Ontology Learningcited by 75
  20. 2021Template-free Prompt Tuning for Few-shot NERcited by 73
  21. 2023Large Language Models and Knowledge Graphs: Opportunities and Challengescited by 71
  22. 2023Corex: Pushing the Boundaries of Complex Reasoning through Multi-Model Collaborationcited by 70
  23. 2023Detecting Edit Failures In Large Language Models: An Improved Specificity Benchmarkcited by 62
  24. 2022Housekeep: Tidying Virtual Households using Commonsense Reasoningcited by 57
  25. 2022Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictionscited by 57
  26. 2023Journey to the Center of the Knowledge Neurons: Discoveries of Language-Independent Knowledge Neurons and Degenerate Knowledge Neuronscited by 56
  27. 2024Context-DPO: Aligning Language Models for Context-Faithfulnesscited by 50
  28. 2023Do Large Language Models Know about Facts?cited by 49
  29. 2022Read before Generate! Faithful Long Form Question Answering with Machine Readingcited by 47
  30. 2021Prompting Contrastive Explanations for Commonsense Reasoning Taskscited by 45
  31. 2022DeepStruct: Pretraining of Language Models for Structure Predictioncited by 45
  32. 2023Large Language Models and Simple, Stupid Bugscited by 45
  33. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 45
  34. 2022No More Fine-Tuning? An Experimental Evaluation of Prompt Tuning in Code Intelligencecited by 41
  35. 2022Exemplar Guided Deep Neural Network for Spatial Transcriptomics Analysis of Gene Expression Predictioncited by 39
  36. 2021Good Examples Make A Faster Learner: Simple Demonstration-based Learning for Low-resource NERcited by 38
  37. 2022Best Prompts for Text-to-Image Models and How to Find Themcited by 35
  38. 2024Small Models, Big Insights: Leveraging Slim Proxy Models To Decide When and What to Retrieve for LLMscited by 34

50 most cited of 181 All 181 (JSON)