mlsanityData to 7 Oct 2026

When Do LLMs Need Retrieval Augmentation? Mitigating LLMs' Overconfidence Helps Retrieval Augmentation

arXiv 2402.11457 ↗First posted 18 Feb 2024cs.CLcited by 59

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have been found to have difficulty knowing they do not possess certain knowledge and tend to provide specious answers in such cases. Retrieval Augmentation (RA) has been extensively studied to mitigate LLMs' hallucinations. However, due to the extra overhead and unassured quality of retrieval, it may not be optimal to conduct RA all the time. A straightforward idea is to only conduct retrieval when LLMs are uncertain about a question. This motivates us to enhance the LLMs' ability to perceive their knowledge boundaries to help RA. In this paper, we first quantitatively measure LLMs' such ability and confirm their overconfidence. Then, we study how LLMs' certainty about a question correlates with their dependence on external retrieved information. We propose several methods to enhance LLMs' perception of knowledge boundaries and show that they are effective in reducing overconfidence. Additionally, equipped with these methods, LLMs can achieve comparable or even better performance of RA with much fewer retrieval calls.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Builds on 23 of its 38 references that are in the map. First posted on arXiv 18 Feb 2024.

Builds on

23 of 38 references.

References · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2017On Calibration of Modern Neural Networkscited by 4,860
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  7. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  8. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  9. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  10. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  11. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  12. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
11 more
  1. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
  2. 2021Revisiting the Calibration of Modern Neural Networkscited by 387
  3. 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 296
  4. 2023Do Large Language Models Know What They Don't Know?cited by 238
  5. 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 175
  6. 2023Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentationcited by 146
  7. 2021End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answeringcited by 133
  8. 2021Semantic Models for the First-stage Retrieval: A Comprehensive Reviewcited by 57
  9. 2021UnitedQA: A Hybrid Approach for Open Domain Question Answeringcited by 42
  10. 2023Black-box Adversarial Attacks against Dense Retrieval Models: A Multi-view Contrastive Learning Methodcited by 29
  11. 2023A Comparative Study of Training Objectives for Clarification Facet Generationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 19; 2025: 27; 2026: 13.

59 papers.

Sort:

By area:

  1. 2024Tool Learning in the Wild: Empowering Language Models as Automatic Tool Agentscited by 56
  2. 2024HtmlRAG: HTML is Better Than Plain Text for Modeling Retrieved Knowledge in RAG Systemscited by 37
  3. 2024DYNAMICQA: Tracing Internal Knowledge Conflicts in Language Modelscited by 34
  4. 2025Adaptive Retrieval Without Self-Knowledge? Bringing Uncertainty Back Homecited by 33
  5. 2025Towards Fully Exploiting LLM Internal States to Enhance Knowledge Boundary Perceptioncited by 26
  6. 2024Are Large Language Models More Honest in Their Probabilistic or Verbalized Confidence?cited by 23
  7. 2024Are Large Language Models Good at Utility Judgments?cited by 22
  8. 2024A Survey on the Honesty of Large Language Modelscited by 21
  9. 2024Perception of Knowledge Boundary for Large Language Models through Semi-open-ended Question Answeringcited by 20
  10. 2024Towards Knowledge Checking in Retrieval-augmented Generation: A Representation Perspectivecited by 20
  11. 2024MINTQA: A Multi-Hop Question Answering Benchmark for Evaluating LLMs on New and Tail Knowledgecited by 20
  12. 2024Is Factuality Enhancement a Free Lunch For LLMs? Better Factuality Can Lead to Worse Context-Faithfulnesscited by 17
38 more
  1. 2025Investigating the Impact of Quantization Methods on the Safety and Reliability of Large Language Modelscited by 17
  2. 2024Reformatted Alignmentcited by 15
  3. 2025Empowering GraphRAG with Knowledge Filtering and Integrationcited by 15
  4. 2024Multi-granular Adversarial Attacks against Black-box Neural Ranking Modelscited by 14
  5. 2024Can Large Language Models Grasp Legal Theories? Enhance Legal Reasoning with Insights from Multi-Agent Collaborationcited by 14
  6. 2025Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missingcited by 14
  7. 2024CtrlA: Adaptive Retrieval-Augmented Generation via Inherent Controlcited by 13
  8. 2025Divide-Then-Align: Honest Alignment based on the Knowledge Boundary of RAGcited by 13
  9. 2025Stop-RAG: Value-Based Retrieval Control for Iterative RAGcited by 13
  10. 2025Do Large Language Models Know What They Are Capable Of?cited by 9
  11. 2026Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantificationcited by 9
  12. 2024A Survey of Calibration Process for Black-Box LLMscited by 8
  13. 2025Focusing by Contrastive Attention: Enhancing VLMs' Visual Reasoningcited by 8
  14. 2025Modeling Uncertainty Trends for Timely Retrieval in Dynamic RAGcited by 8
  15. 2025Line of Duty: Evaluating LLM Self-Knowledge via Consistency in Feasibility Boundariescited by 7
  16. 2025Do LLMs Really Know What They Don't Know? Internal States Mainly Reflect Knowledge Recall Rather Than Truthfulnesscited by 7
  17. 2025Towards Context-Robust LLMs: A Gated Representation Fine-tuning Approachcited by 6
  18. 2025AI Awarenesscited by 6
  19. 2025Accelerating Adaptive Retrieval Augmented Generation via Instruction-Driven Representation Reduction of Retrieval Overlapscited by 6
  20. 2025Popular but Wrong: Understanding and Mitigating LLM Overconfidence through Knowledge Popularitycited by 6
  21. 2026Seeing through the Conflict: Transparent Knowledge Conflict Handling in Retrieval-Augmented Generationcited by 6
  22. 2024Evaluating Implicit Bias in Large Language Models by Attacking From a Psychometric Perspectivecited by 5
  23. 2025KnowRL: Teaching Language Models to Know What They Knowcited by 5
  24. 2026Overconfidence and Calibration in Medical VQA: Empirical Findings and Hallucination-Aware Mitigationcited by 5
  25. 2026Confidence Calibration for Multimodal LLMs: An Empirical Study through Medical VQAcited by 4
  26. 2025Bridging External and Parametric Knowledge: Mitigating Hallucination of LLMs with Shared-Private Semantic Synergy in Dual-Stream Knowledgecited by 3
  27. 2025ConfRAG: Confidence-Guided Retrieval-Augmenting Generationcited by 3
  28. 2025Annotation-Efficient Universal Honesty Alignmentcited by 3
  29. 2026Evaluating and Calibrating LLM Confidence on Questions with Multiple Correct Answerscited by 3
  30. 2024Do You Know What You Are Talking About? Characterizing Query-Knowledge Relevance For Reliable Retrieval Augmented Generationcited by 2
  31. 2025Mirage of Mastery: Memorization Tricks LLMs into Artificially Inflated Self-Knowledgecited by 2
  32. 2026Language-Coupled Reinforcement Learning for Multilingual Retrieval-Augmented Generationcited by 2
  33. 2026Linking Perception, Confidence and Accuracy in MLLMscited by 2
  34. 2025Foundations of GenIRcited by 1
  35. 2025OmniBench-RAG: A Multi-Domain Evaluation Platform for Retrieval-Augmented Generation Toolscited by 1
  36. 2025Detecting Hallucinations in Graph Retrieval-Augmented Generation via Attention Patterns and Semantic Alignmentcited by 1
  37. 2026Understanding the Interplay between LLMs' Utilisation of Parametric and Contextual Knowledge: A keynote at ECIR 2025cited by 1
  38. 2026Process Supervision of Confidence Margin for Calibrated LLM Reasoningcited by 1

50 most cited of 59 All 59 (JSON)