# Classifying and Addressing the Diversity of Errors in Retrieval-Augmented Generation Systems

arXiv: https://arxiv.org/abs/2510.13975 · HTML: https://mlsanity.com/paper/2510.13975/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- Builds on 27 of its 78 references that are in the map.
- First posted on arXiv 15 Oct 2025.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)

## Builds on (most cited first, 27 of 27)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [Retrieval Augmentation Reduces Hallucination in Conversation](https://mlsanity.com/paper/2104.07567/) (2021)
- [Large Dual Encoders Are Generalizable Retrievers](https://mlsanity.com/paper/2112.07899/) (2021)
- [RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs](https://mlsanity.com/paper/2407.02485/) (2024)
- [Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools](https://mlsanity.com/paper/2405.20362/) (2024)
- [Evaluation of Retrieval-Augmented Generation: A Survey](https://mlsanity.com/paper/2405.07437/) (2024)
- [Seven Failure Points When Engineering a Retrieval Augmented Generation System](https://mlsanity.com/paper/2401.05856/) (2024)
- [The Troubling Emergence of Hallucination in Large Language Models -- An Extensive Definition, Quantification, and Prescriptive Remediations](https://mlsanity.com/paper/2310.04988/) (2023)
- [Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting](https://mlsanity.com/paper/2407.08223/) (2024)
- [Walk the Talk? Measuring the Faithfulness of Large Language Model Explanations](https://mlsanity.com/paper/2504.14150/) (2025)
- [Is Semantic Chunking Worth the Computational Cost?](https://mlsanity.com/paper/2410.13070/) (2024)
- [RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework](https://mlsanity.com/paper/2408.01262/) (2024)
- [Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refuse](https://mlsanity.com/paper/2409.11242/) (2024)
- [CLAPNQ: Cohesive Long-form Answers from Passages in Natural Questions for RAG systems](https://mlsanity.com/paper/2404.02103/) (2024)
- [Automated Evaluation of Retrieval-Augmented Language Models with Task-Specific Exam Generation](https://mlsanity.com/paper/2405.13622/) (2024)
- [A Thorough Comparison of Cross-Encoders and LLMs for Reranking SPLADE](https://mlsanity.com/paper/2403.10407/) (2024)
- [Enhancing Q&A Text Retrieval with Ranking Models: Benchmarking, fine-tuning and deploying Rerankers for RAG](https://mlsanity.com/paper/2409.07691/) (2024)
- [Search Engines in an AI Era: The False Promise of Factual and Verifiable Source-Cited Responses](https://mlsanity.com/paper/2410.22349/) (2024)
- [CoFE-RAG: A Comprehensive Full-chain Evaluation Framework for Retrieval-Augmented Generation with Enhanced Data Diversity](https://mlsanity.com/paper/2410.12248/) (2024)
- [Learning to Reason Over Time: Timeline Self-Reflection for Improved Temporal Reasoning in Language Models](https://mlsanity.com/paper/2504.05258/) (2025)

## Built on by (most cited first, 10 of 10)

- [Doctor-RAG: A Failure-Aware Repair Framework for Agentic Retrieval-Augmented Generation](https://mlsanity.com/paper/2604.00865/) (2026)
- [Conformal Agent Error Attribution](https://mlsanity.com/paper/2605.06788/) (2026)
- [Causal methods for LLM development and evaluation](https://mlsanity.com/paper/2605.25998/) (2026)
- [RankJudge: A Multi-Turn LLM-as-a-Judge Synthetic Benchmark Generator](https://mlsanity.com/paper/2605.21748/) (2026)
- [Presentation, Not Mechanism: A Render Confound in Deprecation-Aware Memory Evaluation](https://mlsanity.com/paper/2607.16019/) (2026)
- [M'as contexto no es mejor. Paradoja de la diluci'on vectorial en RAG corporativos](https://mlsanity.com/paper/2601.08851/) (2025)
- [LakeQuest: A Three-Domain Benchmark for Grounded Question Answering across Data Lakes](https://mlsanity.com/paper/2607.12310/) (2026)
- [CIVI: A Framework for Diagnosing Search Agent Failures in Civic Information](https://mlsanity.com/paper/2609.08094/) (2026)
- [Where Does Retrieval-Based Open-Ended Evaluation Fail? Automatic Taxonomy Induction from Long-Form Medical Answer Factuality Verification](https://mlsanity.com/paper/2609.30467/) (2026)
- [Configuring RAG Systems: What Breaks, Where Originates, and How to Fix](https://mlsanity.com/paper/2610.18567/) (2026)

## Abstract (arXiv)

Retrieval-augmented generation (RAG) is a prevalent approach for building LLM-based question-answering systems that can take advantage of external knowledge databases. Due to the complexity of real-world RAG systems, there are many potential causes for erroneous outputs. Understanding the range of errors that can occur in practice is crucial for robust deployment. We present a new taxonomy of the error types that can occur in realistic RAG systems, examples of each, and practical advice for addressing them. Additionally, we curate a dataset of erroneous RAG responses annotated by error types. We then propose an auto-evaluation method aligned with our taxonomy that can be used in practice to track and address errors during development. Code and data are available at https://github.com/layer6ai-labs/rag-error-classification.