mlsanityData to 7 Oct 2026

Seven Failure Points When Engineering a Retrieval Augmented Generation System

arXiv 2401.05856 ↗First posted 11 Jan 2024cs.SE cs.AIcited by 138

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Software engineers are increasingly adding semantic search capabilities to applications using a strategy known as Retrieval Augmented Generation (RAG). A RAG system involves finding documents that semantically match a query and then passing the documents to a large language model (LLM) such as ChatGPT to extract the right answer using an LLM. RAG systems aim to: a) reduce the problem of hallucinated responses from LLMs, b) link sources/references to generated responses, and c) remove the need for annotating documents with meta-data. However, RAG systems suffer from limitations inherent to information retrieval systems and from reliance on LLMs. In this paper, we present an experience report on the failure points of RAG systems from three case studies from separate domains: research, education, and biomedical. We share the lessons learned and present 7 failure points to consider when designing a RAG system. The two key takeaways arising from our work are: 1) validation of a RAG system is only feasible during operation, and 2) the robustness of a RAG system evolves rather than designed in at the start. We conclude with a list of potential research directions on RAG systems for the software engineering community.

Key facts. Cited by 138 papers in the map (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 10 of its 21 references that are in the map. First posted on arXiv 11 Jan 2024.

Builds on

10 of 21 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022FiD-Light: Efficient and Effective Retrieval-Augmented Text Generation28 Sep

Other references · 9

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  4. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  5. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  6. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 307
  7. 2022Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answeringcited by 126
  8. 2020Beware the evolving 'intelligent' web service! An integration architecture tactic to guard AI-first componentscited by 3
  9. 2020Threshy: Supporting Safe Usage of Intelligent Web Servicescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 39; 2025: 58; 2026: 41.

138 papers.

Sort:

By area:

  1. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  2. 2024Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Toolscited by 170
  3. 2024Evaluation of Retrieval-Augmented Generation: A Surveycited by 146
  4. 2024Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?cited by 53
  5. 2024Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Surveycited by 50
  6. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 49
  7. 2024Financial Report Chunking for Effective Retrieval Augmented Generationcited by 46
  8. 2024TC-RAG:Turing-Complete RAG's Case study on Medical LLM Systemscited by 28
  9. 2025Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontierscited by 26
  10. 2024T-RAG: Lessons from the LLM Trenchescited by 23
  11. 2024ConfusedPilot: Confused Deputy Risks in RAG-based LLMscited by 22
  12. 2024HyPA-RAG: A Hybrid Parameter Adaptive Retrieval-Augmented Generation System for AI Legal and Policy Applicationscited by 22
38 more
  1. 2026A Survey on Long-Term Memory Security in LLM Agents: Attacks, Defenses, and Governance Across the Memory Lifecyclecited by 21
  2. 2024When to Retrieve: Teaching LLMs to Utilize Information Retrieval Effectivelycited by 20
  3. 2025FinSage: A Multi-aspect RAG System for Financial Filings Question Answeringcited by 18
  4. 2024LRP4RAG: Detecting Hallucinations in Retrieval-Augmented Generation via Layer-wise Relevance Propagationcited by 17
  5. 2024FACTS About Building Retrieval Augmented Generation-based Chatbotscited by 15
  6. 2024AdapterSwap: Continuous Training of LLMs with Data Removal and Access-Control Guaranteescited by 14
  7. 2024Are LLMs Correctly Integrated into Software Systems?cited by 13
  8. 2025Towards Reliable Retrieval in RAG Systems for Large Legal Datasetscited by 12
  9. 2024Dialectical Alignment: Resolving the Tension of 3H and Security Threats of LLMscited by 11
  10. 2024CommunityKG-RAG: Leveraging Community Structures in Knowledge Graphs for Advanced Retrieval-Augmented Generation in Fact-Checkingcited by 11
  11. 2025Visual RAG: Expanding MLLM visual knowledge without fine-tuningcited by 11
  12. 2024Generating Test Scenarios from NL Requirements using Retrieval-Augmented LLMs: An Industrial Studycited by 9
  13. 2024RE-Adapt: Reverse Engineered Adaptation of Large Language Modelscited by 9
  14. 2025Towards Practical GraphRAG: Efficient Knowledge Graph Construction and Hybrid Retrieval at Scalecited by 9
  15. 2025Classifying and Addressing the Diversity of Errors in Retrieval-Augmented Generation Systemscited by 9
  16. 2024Incorporating External Knowledge and Goal Guidance for LLM-based Conversational Recommender Systemscited by 8
  17. 2024Introducing a new hyper-parameter for RAG: Context Window Utilizationcited by 8
  18. 2025Graph-Augmented Reasoning: Evolving Step-by-Step Knowledge Graph Retrieval for LLM Reasoningcited by 8
  19. 2026InfMem: Learning System-2 Memory Control for Long-Context Agentcited by 8
  20. 2024A RAG Approach for Generating Competency Questions in Ontology Engineeringcited by 7
  21. 2024RAGProbe: An Automated Approach for Evaluating RAG Applicationscited by 7
  22. 2024Optimizing and Evaluating Enterprise Retrieval-Augmented Generation (RAG): A Content Design Perspectivecited by 7
  23. 2024Understanding the Fundamental Design Decisions of Retrieval-Augmented Generation Systemscited by 7
  24. 2025A New HOPE: Domain-agnostic Automatic Evaluation of Text Chunkingcited by 7
  25. 2025GRADE: Generating multi-hop QA and fine-gRAined Difficulty matrix for RAG Evaluationcited by 7
  26. 2024CONFLARE: CONFormal LArge language model REtrievalcited by 6
  27. 2024Contrastive Learning to Improve Retrieval for Real-world Fact Checkingcited by 6
  28. 2024A Methodology for Evaluating RAG Systems: A Case Study On Configuration Dependency Validationcited by 6
  29. 2025RAGVA: Engineering Retrieval Augmented Generation-based Virtual Assistants in Practicecited by 6
  30. 2025Talk Before You Retrieve: Agent-Led Discussions for Better RAG in Medical QAcited by 6
  31. 2025Automatic Dataset Generation for Knowledge Intensive Question Answering Taskscited by 6
  32. 2026Beyond Similarity: Trustworthy Memory Search for Personal AI Agentscited by 6
  33. 2024Development and Testing of a Novel Large Language Model-Based Clinical Decision Support Systems for Medication Safety in 12 Clinical Specialtiescited by 5
  34. 2024Fine-Tuning or Fine-Failing? Debunking Performance Myths in Large Language Modelscited by 5
  35. 2024RAG Foundry: A Framework for Enhancing LLMs for Retrieval Augmented Generationcited by 5
  36. 2024Recent Advances in Attack and Defense Approaches of Large Language Modelscited by 5
  37. 2025POQD: Performance-Oriented Query Decomposer for Multi-vector retrievalcited by 5
  38. 2025A Comprehensive Graph Framework for Question Answering with Mode-Seeking Preference Alignmentcited by 5

50 most cited of 138 All 138 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey
  2. 2024Know Your RAG: Dataset Taxonomy and Generation Strategies for Evaluating RAG Systems
  3. 2024RAG Does Not Work for Enterprises
  4. 2025A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directions
  5. 2024Semantic Tokens in Retrieval Augmented Generation
  6. 2025Retrieval-Augmented Generation in Industry: An Interview Study on Use Cases, Requirements, Challenges, and Evaluation
  7. 2024A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directions
  8. 2025Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Survey