mlsanityData to 7 Oct 2026

The Effect of Scaling, Retrieval Augmentation and Form on the Factual Consistency of Language Models

arXiv 2311.01307 ↗First posted 2 Nov 2023cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) make natural interfaces to factual knowledge, but their usefulness is limited by their tendency to deliver inconsistent answers to semantically equivalent questions. For example, a model might predict both "Anne Redpath passed away in Edinburgh." and "Anne Redpath's life ended in London." In this work, we identify potential causes of inconsistency and evaluate the effectiveness of two mitigation strategies: up-scaling and augmenting the LM with a retrieval corpus. Our results on the LLaMA and Atlas models show that both strategies reduce inconsistency while retrieval augmentation is considerably more efficient. We further consider and disentangle the consistency contributions of different components of Atlas. For all LMs evaluated we find that syntactical form and other evaluation task artifacts impact consistency. Taken together, our results provide a better understanding of the factors affecting the factual consistency of language models.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 26 of its 32 references that are in the map. First posted on arXiv 2 Nov 2023.

Builds on

26 of 32 references.

References · 26

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  5. 2019Language Models as Knowledge Bases?cited by 1,922
  6. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  7. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  8. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  9. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  10. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  11. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  12. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
14 more
  1. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  2. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
  3. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 349
  4. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  5. 2021Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyondcited by 177
  6. 2021Simple Entity-Centric Questions Challenge Dense Retrieverscited by 140
  7. 2021Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Basescited by 107
  8. 2021Measure and Improve Robustness in NLP Models: A Surveycited by 107
  9. 2020When Do You Need Billions of Words of Pretraining Data?cited by 104
  10. 2022TIARA: Multi-grained Retrieval for Robust Question Answering over Large Knowledge Basescited by 59
  11. 2022Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictionscited by 57
  12. 2022An Efficient Memory-Augmented Transformer for Knowledge-Intensive NLP Taskscited by 36
  13. 2021Transferring Knowledge from Vision to Language: How to Achieve it and how to Measure it?cited by 18
  14. 2020Modeling Content and Context with Deep Relational Learningcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 2; 2026: 2.

11 papers.

  1. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  2. 2024From RAGs to rich parameters: Probing how language models utilize external knowledge over parametric information for factual queriescited by 16
  3. 2024Deciphering the Interplay of Parametric and Non-parametric Memory in Retrieval-augmented Language Modelscited by 10
  4. 2024From Form(s) to Meaning: Probing the Semantic Depths of Language Models Using Multisense Consistencycited by 7
  5. 2025CUB: Benchmarking Context Utilisation Techniques for Language Modelscited by 3
  6. 2024UniArk: Improving Generalisation and Consistency for Factual Knowledge Extraction through Debiasingcited by 2
  7. 2024Does Instruction Tuning Make LLMs More Consistent?cited by 2
  8. 2025DS@GT at Touch'e: Large Language Models for Retrieval-Augmented Debatecited by 1
  9. 2024Setting the AI Agenda -- Evidence from Sweden in the ChatGPT Eracited by 0
  10. 2026Retrieval-Augmented Generation in LLMs for Mental Health: Quantifying the Incremental Contribution of Retrieval Within a Layered Safety Architecturecited by 0
  11. 2026DS@GT ARC at Touch'e: Large Language Models for Retrieval-Augmented Debatecited by 0