# Response Quality Assessment for Retrieval-Augmented Generation via Conditional Conformal Factuality

arXiv: https://arxiv.org/abs/2506.20978 · HTML: https://mlsanity.com/paper/2506.20978/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation.
- Builds on 25 of its 33 references that are in the map.
- First posted on arXiv 26 Jun 2025.

## Placed in

- Retrieval-augmented generation / [RAG robustness and adaptive retrieval](https://mlsanity.com/rag/rag-robustness-and-adaptive-retrieval/)

## Builds on (most cited first, 25 of 25)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022)
- [GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models](https://mlsanity.com/paper/2410.05229/) (2024)
- [Ragas: Automated Evaluation of Retrieval Augmented Generation](https://mlsanity.com/paper/2309.15217/) (2023)
- [RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models](https://mlsanity.com/paper/2401.00396/) (2023)
- [ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2311.09476/) (2023)
- [Conformal Language Modeling](https://mlsanity.com/paper/2306.10193/) (2023)
- [Evaluation of Retrieval-Augmented Generation: A Survey](https://mlsanity.com/paper/2405.07437/) (2024)
- [Large language model validity via enhanced conformal prediction methods](https://mlsanity.com/paper/2406.09714/) (2024)
- [Nested conformal prediction and quantile out-of-bag ensemble methods](https://mlsanity.com/paper/1910.10562/) (2019)
- [Trustworthiness in Retrieval-Augmented Generation Systems: A Survey](https://mlsanity.com/paper/2409.10102/) (2024)
- [RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation](https://mlsanity.com/paper/2408.08067/) (2024)
- [Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refuse](https://mlsanity.com/paper/2409.11242/) (2024)
- [C-RAG: Certified Generation Risks for Retrieval-Augmented Language Models](https://mlsanity.com/paper/2402.03181/) (2024)
- [How well do LLMs cite relevant medical references? An evaluation framework and analyses](https://mlsanity.com/paper/2402.02008/) (2024)
- [OLAPH: Improving Factuality in Biomedical Long-form Question Answering](https://mlsanity.com/paper/2405.12701/) (2024)
- [Conformal Prediction Sets Can Cause Disparate Impact](https://mlsanity.com/paper/2410.01888/) (2024)
- [Synchronous Faithfulness Monitoring for Trustworthy Retrieval-Augmented Generation](https://mlsanity.com/paper/2406.13692/) (2024)

## Built on by (most cited first, 8 of 8)

- [Document Summarization with Conformal Importance Guarantees](https://mlsanity.com/paper/2509.20461/) (2025)
- [Principled Context Engineering for RAG: Statistical Guarantees via Conformal Prediction](https://mlsanity.com/paper/2511.17908/) (2025)
- [Federated Language Models Under Bandwidth Budgets: Distillation Rates and Conformal Coverage](https://mlsanity.com/paper/2605.09986/) (2026)
- [Multi-LLM Adaptive Conformal Inference for Reliable LLM Responses](https://mlsanity.com/paper/2602.01285/) (2026)
- [Conformalized Large Language Models under Configuration Shift](https://mlsanity.com/paper/2608.01460/) (2026)
- [Distribution-informed Efficient Conformal Prediction for Full Ranking](https://mlsanity.com/paper/2601.23128/) (2026)
- [Unifying Conformal Language Tasks with In-Context Ensembles](https://mlsanity.com/paper/2609.03005/) (2026)
- [Conformal Factuality Control for Multi-Hop Retrieval-Augmented Generation](https://mlsanity.com/paper/2609.38222/) (2026)

## Abstract (arXiv)

Existing research on Retrieval-Augmented Generation (RAG) primarily focuses on improving overall question-answering accuracy, often overlooking the quality of sub-claims within generated responses. Recent methods that attempt to improve RAG trustworthiness, such as through auto-evaluation metrics, lack probabilistic guarantees or require ground truth answers. To address these limitations, we propose Conformal-RAG, a novel framework inspired by recent applications of conformal prediction (CP) on large language models (LLMs). Conformal-RAG leverages CP and internal information from the RAG mechanism to offer statistical guarantees on response quality. It ensures group-conditional coverage spanning multiple sub-domains without requiring manual labelling of conformal sets, making it suitable for complex RAG applications. Compared to existing RAG auto-evaluation methods, Conformal-RAG offers statistical guarantees on the quality of refined sub-claims, ensuring response reliability without the need for ground truth answers. Additionally, our experiments demonstrate that by leveraging information from the RAG system, Conformal-RAG retains up to 60% more high-quality sub-claims from the response compared to direct applications of CP to LLMs, while maintaining the same reliability guarantee.