mlsanityData to 14 Oct 2026

Classifying and Addressing the Diversity of Errors in Retrieval-Augmented Generation Systems

arXiv 2510.13975 ↗First posted 15 Oct 2025cs.CL cs.LGcited by 10

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Retrieval-augmented generation (RAG) is a prevalent approach for building LLM-based question-answering systems that can take advantage of external knowledge databases. Due to the complexity of real-world RAG systems, there are many potential causes for erroneous outputs. Understanding the range of errors that can occur in practice is crucial for robust deployment. We present a new taxonomy of the error types that can occur in realistic RAG systems, examples of each, and practical advice for addressing them. Additionally, we curate a dataset of erroneous RAG responses annotated by error types. We then propose an auto-evaluation method aligned with our taxonomy that can be used in practice to track and address errors during development. Code and data are available at https://github.com/layer6ai-labs/rag-error-classification.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 27 of its 78 references that are in the map. First posted on arXiv 15 Oct 2025.

Builds on

27 of 78 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024CoFE-RAG: A Comprehensive Full-chain Evaluation Framework for Retrieval-Augmented Generation with Enhanced Data Diversity16 Oct
  2. 2024Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refuse17 Sep
  3. 2024RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework2 Aug
  4. 2024Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting11 Jul
  5. 2024Evaluation of Retrieval-Augmented Generation: A Survey13 May
  6. 2024Seven Failure Points When Engineering a Retrieval Augmented Generation System11 Jan

Other references · 21

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  7. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  8. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  9. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 692
  10. 2021Large Dual Encoders Are Generalizable Retrieverscited by 456
  11. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 202
  12. 2024Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Toolscited by 176
9 more
  1. 2023The Troubling Emergence of Hallucination in Large Language Models -- An Extensive Definition, Quantification, and Prescriptive Remediationscited by 131
  2. 2025Walk the Talk? Measuring the Faithfulness of Large Language Model Explanationscited by 45
  3. 2024Is Semantic Chunking Worth the Computational Cost?cited by 43
  4. 2024CLAPNQ: Cohesive Long-form Answers from Passages in Natural Questions for RAG systemscited by 25
  5. 2024Automated Evaluation of Retrieval-Augmented Language Models with Task-Specific Exam Generationcited by 25
  6. 2024A Thorough Comparison of Cross-Encoders and LLMs for Reranking SPLADEcited by 18
  7. 2024Enhancing Q&A Text Retrieval with Ranking Models: Benchmarking, fine-tuning and deploying Rerankers for RAGcited by 16
  8. 2024Search Engines in an AI Era: The False Promise of Factual and Verifiable Source-Cited Responsescited by 15
  9. 2025Learning to Reason Over Time: Timeline Self-Reflection for Improved Temporal Reasoning in Language Modelscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 9.

10 papers.

  1. 2026Doctor-RAG: A Failure-Aware Repair Framework for Agentic Retrieval-Augmented Generationcited by 5
  2. 2026Conformal Agent Error Attributioncited by 4
  3. 2026Causal methods for LLM development and evaluationcited by 2
  4. 2026RankJudge: A Multi-Turn LLM-as-a-Judge Synthetic Benchmark Generatorcited by 1
  5. 2026Presentation, Not Mechanism: A Render Confound in Deprecation-Aware Memory Evaluationcited by 1
  6. 2025M'as contexto no es mejor. Paradoja de la diluci'on vectorial en RAG corporativoscited by 0
  7. 2026LakeQuest: A Three-Domain Benchmark for Grounded Question Answering across Data Lakescited by 0
  8. 2026CIVI: A Framework for Diagnosing Search Agent Failures in Civic Informationcited by 0
  9. 2026Where Does Retrieval-Based Open-Ended Evaluation Fail? Automatic Taxonomy Induction from Long-Form Medical Answer Factuality Verificationcited by 0
  10. 2026Configuring RAG Systems: What Breaks, Where Originates, and How to Fixcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Know Your RAG: Dataset Taxonomy and Generation Strategies for Evaluating RAG Systems
  2. 2025A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directions
  3. 2024Semantic Tokens in Retrieval Augmented Generation
  4. 2024A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directions
  5. 2024Optimizing and Evaluating Enterprise Retrieval-Augmented Generation (RAG): A Content Design Perspective
  6. 2024Retrieval Augmented Generation Systems: Automatic Dataset Creation, Evaluation and Boolean Agent Setup
  7. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey
  8. 2026A Systems-Level Analysis of Sensitivity, Robustness, and Stability in Retrieval-Augmented Generation