# Configuring RAG Systems: What Breaks, Where Originates, and How to Fix

arXiv: https://arxiv.org/abs/2610.18567 · HTML: https://mlsanity.com/paper/2610.18567/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- Builds on 25 of its 57 references that are in the map.
- First posted on arXiv 13 Oct 2026.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)

## Builds on (most cited first, 25 of 25)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2405.06211/) (2024)
- [A Survey of Hallucination in Large Foundation Models](https://mlsanity.com/paper/2309.05922/) (2023)
- [Seven Failure Points When Engineering a Retrieval Augmented Generation System](https://mlsanity.com/paper/2401.05856/) (2024)
- [Modular RAG: Transforming RAG Systems into LEGO-like Reconfigurable Frameworks](https://mlsanity.com/paper/2407.21059/) (2024)
- [A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directions](https://mlsanity.com/paper/2507.18910/) (2025)
- [Multi-Objectivizing Software Configuration Tuning (for a single performance concern)](https://mlsanity.com/paper/2106.01331/) (2021)
- [Enhancing Retrieval-Augmented Generation: A Study of Best Practices](https://mlsanity.com/paper/2501.07391/) (2025)
- [Hallucination Detection and Hallucination Mitigation: An Investigation](https://mlsanity.com/paper/2401.08358/) (2024)
- [MMO: Meta Multi-Objectivization for Software Configuration Tuning](https://mlsanity.com/paper/2112.07303/) (2021)
- [An Empirical Study on Challenges for LLM Application Developers](https://mlsanity.com/paper/2408.05002/) (2024)
- [METIS: Fast Quality-Aware RAG Systems with Configuration Adaptation](https://mlsanity.com/paper/2412.10543/) (2024)
- [Are LLMs Correctly Integrated into Software Systems?](https://mlsanity.com/paper/2407.05138/) (2024)
- [Distilled Lifelong Self-Adaptation for Configurable Systems](https://mlsanity.com/paper/2501.00840/) (2025)
- [Faster, Cheaper, Better: Multi-Objective Hyperparameter Optimization for LLM and RAG Systems](https://mlsanity.com/paper/2502.18635/) (2025)
- [PromiseTune: Unveiling Causally Promising and Explainable Configuration Tuning](https://mlsanity.com/paper/2507.05995/) (2025)
- [Accuracy Can Lie: On the Impact of Surrogate Model in Configuration Tuning](https://mlsanity.com/paper/2501.01876/) (2025)
- [Classifying and Addressing the Diversity of Errors in Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2510.13975/) (2025)
- [Dually Hierarchical Drift Adaptation for Online Configuration Performance Learning](https://mlsanity.com/paper/2507.08730/) (2025)
- [Towards offensive language detection and reduction in four Software Engineering communities](https://mlsanity.com/paper/2106.02245/) (2021)
- [On Reporting Performance and Accuracy Bugs for Deep Learning Frameworks: An Exploratory Study from GitHub](https://mlsanity.com/paper/2204.07893/) (2022)
- [Revealing Domain-Spatiality Patterns for Configuration Tuning: Domain Knowledge Meets Fitness Landscapes](https://mlsanity.com/paper/2603.19897/) (2026)
- [Evaluating Useful Surrogate Models for Configuration Tuning Beyond Accuracy: A Fitness Landscape Analysis Perspective](https://mlsanity.com/paper/2509.21945/) (2025)
- [CDS4RAG: Cyclic Dual-Sequential Hyperparameter Optimization for RAG](https://mlsanity.com/paper/2605.08333/) (2026)

## Abstract (arXiv)

Retrieval-Augmented Generation (RAG) has emerged as a dominant paradigm for improving Large Language Model (LLM) by intertwining it with other components, such as a vector/graph database. Therefore, the resulting RAG systems are often highly configurable, under which a non-trivial amount of failures are related to the configurations. However, there remains a lack of systematic understanding regarding the configuration issues for RAG systems. To bridge this gap, in this paper, we conduct the first large-scale empirical study of 654 real-world configuration issues collected from 10 diverse RAG systems. Drawing on this, we provide a complete categorization of the symptoms, root causes, relevant stages, and fix patterns of configuration issues for RAG systems, as well as several new observations. Our findings not only reveal the characteristics of configuration issues, but also strongly link them to the unique pipeline stages of RAG, e.g., we found that most symptoms can be observed at the early or late stage, while the stage of the root cause might not be at the same stage as where the symptom is identified. We also provide nine actionable insights to the practitioners on testing, localizing, and fixing the configuration issues for RAG systems.