mlsanityData to 14 Oct 2026

Engineering RAG Systems for Real-World Applications: Design, Development, and Evaluation

arXiv 2506.20869 ↗First posted 25 Jun 2025cs.SE cs.AI cs.IRcited by 3

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Retrieval-Augmented Generation (RAG) systems are emerging as a key approach for grounding Large Language Models (LLMs) in external knowledge, addressing limitations in factual accuracy and contextual relevance. However, there is a lack of empirical studies that report on the development of RAG-based implementations grounded in real-world use cases, evaluated through general user involvement, and accompanied by systematic documentation of lessons learned. This paper presents five domain-specific RAG applications developed for real-world scenarios across governance, cybersecurity, agriculture, industrial research, and medical diagnostics. Each system incorporates multilingual OCR, semantic retrieval via vector embeddings, and domain-adapted LLMs, deployed through local servers or cloud APIs to meet distinct user needs. A web-based evaluation involving a total of 100 participants assessed the systems across six dimensions: (i) Ease of Use, (ii) Relevance, (iii) Transparency, (iv) Responsiveness, (v) Accuracy, and (vi) Likelihood of Recommendation. Based on user feedback and our development experience, we documented twelve key lessons learned, highlighting technical, operational, and ethical challenges affecting the reliability and usability of RAG systems in practice.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 28 of its 36 references that are in the map. First posted on arXiv 25 Jun 2025.

Builds on

28 of 36 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Enhancing Retrieval-Augmented Generation: A Study of Best Practices13 Jan
  2. 2024Advanced System Integration: Analyzing OpenAPI Chunking for Retrieval-Augmented Generation29 Nov
  3. 2024Know Your RAG: Dataset Taxonomy and Generation Strategies for Evaluating RAG Systems29 Nov
  4. 2024Towards Knowledge Checking in Retrieval-augmented Generation: A Representation Perspective21 Nov
  5. 2024Developing Retrieval Augmented Generation (RAG) based LLM Systems from PDFs: An Experience Report21 Oct
  6. 2024A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directions3 Oct
  7. 2024Retrieval-augmented generation in multilingual settings1 Jul
  8. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models10 May
  9. 2024Retrieval-Augmented Generation for AI-Generated Content: A Survey29 Feb
  10. 2024Corrective Retrieval Augmented Generation29 Jan
  11. 2024Seven Failure Points When Engineering a Retrieval Augmented Generation System11 Jan

Other references · 17

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  3. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  4. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 338
  5. 2024LightRAG: Simple and Fast Retrieval-Augmented Generationcited by 315
  6. 2025Agentic Retrieval-Augmented Generation: A Survey on Agentic RAGcited by 249
  7. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 245
  8. 2024Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generationcited by 162
  9. 2024HybridRAG: Integrating Knowledge Graphs and Vector Retrieval Augmented Generation for Efficient Information Extractioncited by 93
  10. 2024Searching for Best Practices in Retrieval-Augmented Generationcited by 83
  11. 2024OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generationcited by 26
  12. 2023Generative Artificial Intelligence for Software Engineering -- A Research Agendacited by 23
5 more
  1. 2024Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documentscited by 17
  2. 2024Ask-EDA: A Design Assistant Empowered by LLM, Hybrid RAG and Abbreviation De-hallucinationcited by 14
  3. 2024Towards Interpretable Radiology Report Generation via Concept Bottlenecks using a Multi-Agentic RAGcited by 9
  4. 2024StackRAG Agent: Improving Developer Answers with Retrieval-Augmented Generationcited by 1
  5. 2024Path-RAG: Knowledge-Guided Key Region Retrieval for Open-ended Pathology Visual Question Answeringcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026GraphRAG for Engineering Diagrams: ChatP&ID Enables LLM Interaction with P&IDscited by 1
  2. 2026Towards AI Evaluation in Domain-Specific RAG Systems: The AgriHubi Case Studycited by 0
  3. 2026Context Before Code: An Experience Report on Vibe Coding in Practicecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directions
  2. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey
  3. 2023Retrieval-Augmented Generation for Large Language Models: A Survey
  4. 2024Deploying Large Language Models With Retrieval Augmented Generation
  5. 2025Retrieval-Augmented Generation in Industry: An Interview Study on Use Cases, Requirements, Challenges, and Evaluation
  6. 2025When Retrieval Succeeds and Fails: Rethinking Retrieval-Augmented Generation for LLMs
  7. 2025Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Survey
  8. 2026Pursuing Best Industrial Practices for Retrieval-Augmented Generation in the Medical Domain