mlsanityData to 14 Oct 2026

Reliable Decision Making via Calibration Oriented Retrieval Augmented Generation

arXiv 2411.08891 ↗First posted 28 Oct 2024cs.IR cs.AIcited by 6

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Recently, Large Language Models (LLMs) have been increasingly used to support various decision-making tasks, assisting humans in making informed decisions. However, when LLMs confidently provide incorrect information, it can lead humans to make suboptimal decisions. To prevent LLMs from generating incorrect information on topics they are unsure of and to improve the accuracy of generated content, prior works have proposed Retrieval Augmented Generation (RAG), where external documents are referenced to generate responses. However, previous RAG methods focus only on retrieving documents most relevant to the input query, without specifically aiming to ensure that the human user's decisions are well-calibrated. To address this limitation, we propose a novel retrieval method called Calibrated Retrieval-Augmented Generation (CalibRAG), which ensures that decisions informed by RAG are well-calibrated. Then we empirically validate that CalibRAG improves calibration performance as well as accuracy, compared to other baselines across various datasets.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 41 of its 61 references that are in the map. First posted on arXiv 28 Oct 2024.

Builds on

41 of 61 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022A Survey on Retrieval-Augmented Text Generation2 Feb

Other references · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2024The Llama 3 Herd of Modelscited by 14,715
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  8. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  9. 2017On Calibration of Modern Neural Networkscited by 4,960
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  12. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
28 more
  1. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  2. 2023Mistral 7Bcited by 3,164
  3. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  4. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  5. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  6. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  7. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  8. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  9. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  10. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  11. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 692
  12. 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 422
  13. 2021Revisiting the Calibration of Modern Neural Networkscited by 390
  14. 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 364
  15. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 360
  16. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 331
  17. 2020Distilling Knowledge from Reader to Retriever for Question Answeringcited by 226
  18. 2023RA-DIT: Retrieval-Augmented Dual Instruction Tuningcited by 195
  19. 2023MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrievalcited by 185
  20. 2019Evaluating model calibration in classificationcited by 168
  21. 2019Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Trainingcited by 110
  22. 2021WebQA: Multihop and Multimodal QAcited by 109
  23. 2024Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertaintycited by 106
  24. 2024Enhancing LLM Factual Accuracy with RAG to Counter Hallucinations: A Case Study on Domain-Specific Queries in Private Knowledge-Basescited by 77
  25. 2024Linguistic Calibration of Long-Form Generationscited by 60
  26. 2023Fine-Tuning Language Models with Advantage-Induced Policy Alignmentcited by 48
  27. 2022Faithfulness in Natural Language Generation: A Systematic Survey of Analysis, Evaluation and Optimization Methodscited by 27
  28. 2022Improving Bilingual Lexicon Induction with Cross-Encoder Rerankingcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2025Verbalized Confidence Triggers Self-Verification: Emergent Behavior Without Explicit Reasoning Supervisioncited by 9
  2. 2026Learning When to Remember: Risk-Sensitive Contextual Bandits for Abstention-Aware Memory Retrieval in LLM-Based Coding Agentscited by 3
  3. 2026Evaluating Confidence-Gated Retrieval with Matched Trajectory Replaycited by 3
  4. 2026Doc2AHP: Inferring Structured Multi-Criteria Decision Models via Semantic Trees with LLMscited by 1
  5. 2026Bridging the Missing-Modality Gap: Improving Text-Only Calibration of Vision Language Modelscited by 0
  6. 2026Latent Confidence Alignment for LLM Self-Assessmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SGIC: A Self-Guided Iterative Calibration Framework for RAG
  2. 2026Know Before You Fetch: Calibrated Retrieval-Budget Allocation for Retrieval-Augmented Generation
  3. 2024Corrective Retrieval Augmented Generation
  4. 2025Investigating the Robustness of Retrieval-Augmented Generation at the Query Level
  5. 2024Retrieval-Augmented Generation with Estimation of Source Reliability
  6. 2024Understanding the Impact of Confidence in Retrieval Augmented Generation: A Case Study in the Medical Domain
  7. 2024Deploying Large Language Models With Retrieval Augmented Generation
  8. 2025Fast or Better? Balancing Accuracy and Cost in Retrieval-Augmented Generation with Flexible User Control