# When Do LLMs Need Retrieval Augmentation? Mitigating LLMs' Overconfidence Helps Retrieval Augmentation

arXiv: https://arxiv.org/abs/2402.11457 · HTML: https://mlsanity.com/paper/2402.11457/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 38 references that are in the map.
- First posted on arXiv 18 Feb 2024.

## Builds on (most cited first, 23 of 23)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [REPLUG: Retrieval-Augmented Black-Box Language Models](https://mlsanity.com/paper/2301.12652/) (2023)
- [How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering](https://mlsanity.com/paper/2012.00955/) (2020)
- [RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2010.08191/) (2020)
- [Revisiting the Calibration of Modern Neural Networks](https://mlsanity.com/paper/2106.07998/) (2021)
- [Generate rather than Retrieve: Large Language Models are Strong Context Generators](https://mlsanity.com/paper/2209.10063/) (2022)
- [Do Large Language Models Know What They Don't Know?](https://mlsanity.com/paper/2305.18153/) (2023)
- [RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking](https://mlsanity.com/paper/2110.07367/) (2021)
- [Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation](https://mlsanity.com/paper/2307.11019/) (2023)
- [End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering](https://mlsanity.com/paper/2106.05346/) (2021)
- [Semantic Models for the First-stage Retrieval: A Comprehensive Review](https://mlsanity.com/paper/2103.04831/) (2021)
- [UnitedQA: A Hybrid Approach for Open Domain Question Answering](https://mlsanity.com/paper/2101.00178/) (2021)
- [Black-box Adversarial Attacks against Dense Retrieval Models: A Multi-view Contrastive Learning Method](https://mlsanity.com/paper/2308.09861/) (2023)
- [A Comparative Study of Training Objectives for Clarification Facet Generation](https://mlsanity.com/paper/2310.00703/) (2023)

## Built on by (most cited first, 30 of 59)

- [Tool Learning in the Wild: Empowering Language Models as Automatic Tool Agents](https://mlsanity.com/paper/2405.16533/) (2024)
- [HtmlRAG: HTML is Better Than Plain Text for Modeling Retrieved Knowledge in RAG Systems](https://mlsanity.com/paper/2411.02959/) (2024)
- [DYNAMICQA: Tracing Internal Knowledge Conflicts in Language Models](https://mlsanity.com/paper/2407.17023/) (2024)
- [Adaptive Retrieval Without Self-Knowledge? Bringing Uncertainty Back Home](https://mlsanity.com/paper/2501.12835/) (2025)
- [Towards Fully Exploiting LLM Internal States to Enhance Knowledge Boundary Perception](https://mlsanity.com/paper/2502.11677/) (2025)
- [Are Large Language Models More Honest in Their Probabilistic or Verbalized Confidence?](https://mlsanity.com/paper/2408.09773/) (2024)
- [A Survey on the Honesty of Large Language Models](https://mlsanity.com/paper/2409.18786/) (2024)
- [Are Large Language Models Good at Utility Judgments?](https://mlsanity.com/paper/2403.19216/) (2024)
- [Perception of Knowledge Boundary for Large Language Models through Semi-open-ended Question Answering](https://mlsanity.com/paper/2405.14383/) (2024)
- [Towards Knowledge Checking in Retrieval-augmented Generation: A Representation Perspective](https://mlsanity.com/paper/2411.14572/) (2024)
- [MINTQA: A Multi-Hop Question Answering Benchmark for Evaluating LLMs on New and Tail Knowledge](https://mlsanity.com/paper/2412.17032/) (2024)
- [Investigating the Impact of Quantization Methods on the Safety and Reliability of Large Language Models](https://mlsanity.com/paper/2502.15799/) (2025)
- [Is Factuality Enhancement a Free Lunch For LLMs? Better Factuality Can Lead to Worse Context-Faithfulness](https://mlsanity.com/paper/2404.00216/) (2024)
- [Reformatted Alignment](https://mlsanity.com/paper/2402.12219/) (2024)
- [Empowering GraphRAG with Knowledge Filtering and Integration](https://mlsanity.com/paper/2503.13804/) (2025)
- [Multi-granular Adversarial Attacks against Black-box Neural Ranking Models](https://mlsanity.com/paper/2404.01574/) (2024)
- [Can Large Language Models Grasp Legal Theories? Enhance Legal Reasoning with Insights from Multi-Agent Collaboration](https://mlsanity.com/paper/2410.02507/) (2024)
- [Divide-Then-Align: Honest Alignment based on the Knowledge Boundary of RAG](https://mlsanity.com/paper/2505.20871/) (2025)
- [Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missing](https://mlsanity.com/paper/2507.10587/) (2025)
- [Stop-RAG: Value-Based Retrieval Control for Iterative RAG](https://mlsanity.com/paper/2510.14337/) (2025)
- [CtrlA: Adaptive Retrieval-Augmented Generation via Inherent Control](https://mlsanity.com/paper/2405.18727/) (2024)
- [Do LLMs Really Know What They Don't Know? Internal States Mainly Reflect Knowledge Recall Rather Than Truthfulness](https://mlsanity.com/paper/2510.09033/) (2025)
- [Do Large Language Models Know What They Are Capable Of?](https://mlsanity.com/paper/2512.24661/) (2025)
- [Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantification](https://mlsanity.com/paper/2604.17112/) (2026)
- [A Survey of Calibration Process for Black-Box LLMs](https://mlsanity.com/paper/2412.12767/) (2024)
- [Focusing by Contrastive Attention: Enhancing VLMs' Visual Reasoning](https://mlsanity.com/paper/2509.06461/) (2025)
- [Modeling Uncertainty Trends for Timely Retrieval in Dynamic RAG](https://mlsanity.com/paper/2511.09980/) (2025)
- [Line of Duty: Evaluating LLM Self-Knowledge via Consistency in Feasibility Boundaries](https://mlsanity.com/paper/2503.11256/) (2025)
- [AI Awareness](https://mlsanity.com/paper/2504.20084/) (2025)
- [Evaluating Implicit Bias in Large Language Models by Attacking From a Psychometric Perspective](https://mlsanity.com/paper/2406.14023/) (2024)

## Abstract (arXiv)

Large Language Models (LLMs) have been found to have difficulty knowing they do not possess certain knowledge and tend to provide specious answers in such cases. Retrieval Augmentation (RA) has been extensively studied to mitigate LLMs' hallucinations. However, due to the extra overhead and unassured quality of retrieval, it may not be optimal to conduct RA all the time. A straightforward idea is to only conduct retrieval when LLMs are uncertain about a question. This motivates us to enhance the LLMs' ability to perceive their knowledge boundaries to help RA. In this paper, we first quantitatively measure LLMs' such ability and confirm their overconfidence. Then, we study how LLMs' certainty about a question correlates with their dependence on external retrieved information. We propose several methods to enhance LLMs' perception of knowledge boundaries and show that they are effective in reducing overconfidence. Additionally, equipped with these methods, LLMs can achieve comparable or even better performance of RA with much fewer retrieval calls.