mlsanityData to 14 Oct 2026

Unanswerability Evaluation for Retrieval Augmented Generation

arXiv 2412.12300 ↗First posted 16 Dec 2024cs.CLcited by 4

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Existing evaluation frameworks for retrieval-augmented generation (RAG) systems focus on answerable queries, but they overlook the importance of appropriately rejecting unanswerable requests. In this paper, we introduce UAEval4RAG, a framework designed to evaluate whether RAG systems can handle unanswerable queries effectively. We define a taxonomy with six unanswerable categories, and UAEval4RAG automatically synthesizes diverse and challenging queries for any given knowledge base with unanswered ratio and acceptable ratio metrics. We conduct experiments with various RAG components, including retrieval models, rewriting methods, rerankers, language models, and prompting strategies, and reveal hidden trade-offs in performance of RAG systems. Our findings highlight the critical role of component selection and prompt design in optimizing RAG systems to balance the accuracy of answerable queries with high rejection rates of unanswerable ones. UAEval4RAG provides valuable insights and tools for developing more robust and reliable RAG systems.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 31 of its 43 references that are in the map. First posted on arXiv 16 Dec 2024.

Builds on

31 of 43 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Evaluation of Retrieval-Augmented Generation: A Survey13 May
  2. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries27 Jan
  3. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems16 Nov

Other references · 28

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  4. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  5. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  6. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  7. 2024RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrievalcited by 532
  8. 2022Precise Zero-Shot Dense Retrieval without Relevance Labelscited by 517
  9. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
  10. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 390
  11. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 333
  12. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 285
16 more
  1. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 200
  2. 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 180
  3. 2020UnQovering Stereotyping Biases via Underspecified Questionscited by 126
  4. 2022Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectlycited by 89
  5. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedbackcited by 87
  6. 2024FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"cited by 86
  7. 2022Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Surveycited by 80
  8. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safetycited by 77
  9. 2022CREPE: Open-Domain Question Answering with False Presuppositionscited by 72
  10. 2019Question Answering for Privacy Policies: Combining Computational and Legal Perspectivescited by 67
  11. 2023Uncertainty in Natural Language Generation: From Theory to Applicationscited by 61
  12. 2022(QA)2: Question Answering with Questionable Assumptionscited by 51
  13. 2020Challenges in Information-Seeking QA: Unanswerable Questions and Paragraph Retrievalcited by 44
  14. 2023Beyond the Safeguards: Exploring the Security Risks of ChatGPTcited by 36
  15. 2023Assessing Language Model Deployment with Risk Cardscited by 30
  16. 2023ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented Large Language Models via Transferable Adversarial Attackscited by 28

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAGcited by 21
  2. 2026KNIGHT: Knowledge Graph-Driven Multiple-Choice Question Generation with Adaptive Hardness Calibrationcited by 4
  3. 2025Investigating Retrieval-Augmented Generation Systems on Unanswerable, Uncheatable, Realistic, Multi-hop Queriescited by 2
  4. 2026RAGExplorer: A Visual Analytics System for the Comparative Diagnosis of RAG Systemscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025Investigating the Robustness of Retrieval-Augmented Generation at the Query Level
  2. 2024Semantic Tokens in Retrieval Augmented Generation
  3. 2025Enhancing Retrieval-Augmented Generation: A Study of Best Practices
  4. 2025Classifying and Addressing the Diversity of Errors in Retrieval-Augmented Generation Systems
  5. 2024Seven Failure Points When Engineering a Retrieval Augmented Generation System
  6. 2025mmRAG: A Modular Benchmark for Retrieval-Augmented Generation over Text, Tables, and Knowledge Graphs
  7. 2025A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directions
  8. 2025Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontiers