mlsanityData to 14 Oct 2026

Configuring RAG Systems: What Breaks, Where Originates, and How to Fix

arXiv 2610.18567 ↗First posted 13 Oct 2026cs.AIcited by 0

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Retrieval-Augmented Generation (RAG) has emerged as a dominant paradigm for improving Large Language Model (LLM) by intertwining it with other components, such as a vector/graph database. Therefore, the resulting RAG systems are often highly configurable, under which a non-trivial amount of failures are related to the configurations. However, there remains a lack of systematic understanding regarding the configuration issues for RAG systems. To bridge this gap, in this paper, we conduct the first large-scale empirical study of 654 real-world configuration issues collected from 10 diverse RAG systems. Drawing on this, we provide a complete categorization of the symptoms, root causes, relevant stages, and fix patterns of configuration issues for RAG systems, as well as several new observations. Our findings not only reveal the characteristics of configuration issues, but also strongly link them to the unique pipeline stages of RAG, e.g., we found that most symptoms can be observed at the early or late stage, while the stage of the root cause might not be at the same stage as where the symptom is identified. We also provide nine actionable insights to the practitioners on testing, localizing, and fixing the configuration issues for RAG systems.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 25 of its 57 references that are in the map. First posted on arXiv 13 Oct 2026.

Builds on

25 of 57 references.

Closest ancestors · 9

Same direction, newest first

  1. 2026CDS4RAG: Cyclic Dual-Sequential Hyperparameter Optimization for RAG8 May
  2. 2025Classifying and Addressing the Diversity of Errors in Retrieval-Augmented Generation Systems15 Oct
  3. 2025A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directions25 Jul
  4. 2025Faster, Cheaper, Better: Multi-Objective Hyperparameter Optimization for LLM and RAG Systems25 Feb
  5. 2025Enhancing Retrieval-Augmented Generation: A Study of Best Practices13 Jan
  6. 2024METIS: Fast Quality-Aware RAG Systems with Configuration Adaptation13 Dec
  7. 2024Modular RAG: Transforming RAG Systems into LEGO-like Reconfigurable Frameworks26 Jul
  8. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models10 May
  9. 2024Seven Failure Points When Engineering a Retrieval Augmented Generation System11 Jan

Other references · 16

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  3. 2023A Survey of Hallucination in Large Foundation Modelscited by 368
  4. 2021Multi-Objectivizing Software Configuration Tuning (for a single performance concern)cited by 28
  5. 2024Hallucination Detection and Hallucination Mitigation: An Investigationcited by 21
  6. 2021MMO: Meta Multi-Objectivization for Software Configuration Tuningcited by 20
  7. 2024An Empirical Study on Challenges for LLM Application Developerscited by 18
  8. 2024Are LLMs Correctly Integrated into Software Systems?cited by 15
  9. 2025Distilled Lifelong Self-Adaptation for Configurable Systemscited by 12
  10. 2025PromiseTune: Unveiling Causally Promising and Explainable Configuration Tuningcited by 12
  11. 2025Accuracy Can Lie: On the Impact of Surrogate Model in Configuration Tuningcited by 10
  12. 2025Dually Hierarchical Drift Adaptation for Online Configuration Performance Learningcited by 8
4 more
  1. 2021Towards offensive language detection and reduction in four Software Engineering communitiescited by 7
  2. 2022On Reporting Performance and Accuracy Bugs for Deep Learning Frameworks: An Exploratory Study from GitHubcited by 7
  3. 2026Revealing Domain-Spatiality Patterns for Configuration Tuning: Domain Knowledge Meets Fitness Landscapescited by 5
  4. 2025Evaluating Useful Surrogate Models for Configuration Tuning Beyond Accuracy: A Fitness Landscape Analysis Perspectivecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024A Methodology for Evaluating RAG Systems: A Case Study On Configuration Dependency Validation
  2. 2024RAG Foundry: A Framework for Enhancing LLMs for Retrieval Augmented Generation
  3. 2024XRAG: eXamining the Core -- Benchmarking Foundational Components in Advanced Retrieval-Augmented Generation
  4. 2024InspectorRAGet: An Introspection Platform for RAG Evaluation
  5. 2025A System for Comprehensive Assessment of RAG Frameworks
  6. 2026Mapping the RAG Landscape: A Four Axis Taxonomy of Efficiency, Defense, Interactivity, and Reasoning
  7. 2023Retrieval-Augmented Generation for Large Language Models: A Survey
  8. 2024Towards Understanding Systems Trade-offs in Retrieval-Augmented Generation Model Inference