mlsanityData to 14 Oct 2026

CONFLARE: CONFormal LArge language model REtrieval

arXiv 2404.04287 ↗First posted 4 Apr 2024cs.CL cs.AIcited by 6

On the map: Retrieval-augmented generation → RAG robustness and adaptive retrieval

Abstract (arXiv)

Retrieval-augmented generation (RAG) frameworks enable large language models (LLMs) to retrieve relevant information from a knowledge base and incorporate it into the context for generating responses. This mitigates hallucinations and allows for the updating of knowledge without retraining the LLM. However, RAG does not guarantee valid responses if retrieval fails to identify the necessary information as the context for response generation. Also, if there is contradictory content, the RAG response will likely reflect only one of the two possible responses. Therefore, quantifying uncertainty in the retrieval process is crucial for ensuring RAG trustworthiness. In this report, we introduce a four-step framework for applying conformal prediction to quantify retrieval uncertainty in RAG frameworks. First, a calibration set of questions answerable from the knowledge base is constructed. Each question's embedding is compared against document embeddings to identify the most relevant document chunks containing the answer and record their similarity scores. Given a user-specified error rate (α), these similarity scores are then analyzed to determine a similarity score cutoff threshold. During inference, all chunks with similarity exceeding this threshold are retrieved to provide context to the LLM, ensuring the true answer is captured in the context with a (1-α) confidence level. We provide a Python package that enables users to implement the entire workflow proposed in our work, only using LLMs and without human intervention.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation. Builds on 13 of its 19 references that are in the map. First posted on arXiv 4 Apr 2024.

Builds on

13 of 19 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024C-RAG: Certified Generation Risks for Retrieval-Augmented Language Models5 Feb

Other references · 12

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  3. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 792
  4. 2023Improving Text Embeddings with Large Language Modelscited by 303
  5. 2022Conformal Risk Controlcited by 299
  6. 2023Conformal Language Modelingcited by 170
  7. 2023Conformal Prediction with Large Language Models for Multi-Choice Question Answeringcited by 139
  8. 2024Seven Failure Points When Engineering a Retrieval Augmented Generation Systemcited by 139
  9. 2020Conformal Prediction: a Unified Review of Theory and New Challengescited by 118
  10. 2024Large language models surpass human experts in predicting neuroscience resultscited by 69
  11. 2023Does GPT-4 surpass human performance in linguistic pragmatics?cited by 16
  12. 2019A Comparison of Semantic Similarity Methods for Maximum Human Interpretabilitycited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 2.

6 papers.

  1. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 121
  2. 2025Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Surveycited by 30
  3. 2025Conformal Prediction Sets for Deep Generative Models via Reduction to Conformal Regressioncited by 10
  4. 2025Principled Context Engineering for RAG: Statistical Guarantees via Conformal Predictioncited by 6
  5. 2026Inference-Time Conformal Reasoning with Valid Factuality Control for Large Language Modelscited by 3
  6. 2026Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Studycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026URAG: A Benchmark for Uncertainty Quantification in Retrieval-Augmented Large Language Models
  2. 2025Response Quality Assessment for Retrieval-Augmented Generation via Conditional Conformal Factuality
  3. 2023Retrieval-Augmented Generation for Large Language Models: A Survey
  4. 2024Contextual Compression in Retrieval-Augmented Generation for Large Language Models: A Survey
  5. 2024Astute RAG: Overcoming Imperfect Retrieval Augmentation and Knowledge Conflicts for Large Language Models
  6. 2025Evaluating the Retrieval Robustness of Large Language Models
  7. 2024Do You Know What You Are Talking About? Characterizing Query-Knowledge Relevance For Reliable Retrieval Augmented Generation
  8. 2025Investigating the Robustness of Retrieval-Augmented Generation at the Query Level