# Engineering RAG Systems for Real-World Applications: Design, Development, and Evaluation

arXiv: https://arxiv.org/abs/2506.20869 · HTML: https://mlsanity.com/paper/2506.20869/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- Builds on 28 of its 36 references that are in the map.
- First posted on arXiv 25 Jun 2025.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)

## Builds on (most cited first, 28 of 28)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2405.06211/) (2024)
- [Retrieval-Augmented Generation for AI-Generated Content: A Survey](https://mlsanity.com/paper/2402.19473/) (2024)
- [AgentTuning: Enabling Generalized Agent Abilities for LLMs](https://mlsanity.com/paper/2310.12823/) (2023)
- [LightRAG: Simple and Fast Retrieval-Augmented Generation](https://mlsanity.com/paper/2410.05779/) (2024)
- [Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG](https://mlsanity.com/paper/2501.09136/) (2025)
- [VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents](https://mlsanity.com/paper/2410.10594/) (2024)
- [Corrective Retrieval Augmented Generation](https://mlsanity.com/paper/2401.15884/) (2024)
- [Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation](https://mlsanity.com/paper/2409.12941/) (2024)
- [Seven Failure Points When Engineering a Retrieval Augmented Generation System](https://mlsanity.com/paper/2401.05856/) (2024)
- [HybridRAG: Integrating Knowledge Graphs and Vector Retrieval Augmented Generation for Efficient Information Extraction](https://mlsanity.com/paper/2408.04948/) (2024)
- [Searching for Best Practices in Retrieval-Augmented Generation](https://mlsanity.com/paper/2407.01219/) (2024)
- [A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directions](https://mlsanity.com/paper/2410.12837/) (2024)
- [Retrieval-augmented generation in multilingual settings](https://mlsanity.com/paper/2407.01463/) (2024)
- [OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generation](https://mlsanity.com/paper/2412.02592/) (2024)
- [Enhancing Retrieval-Augmented Generation: A Study of Best Practices](https://mlsanity.com/paper/2501.07391/) (2025)
- [Generative Artificial Intelligence for Software Engineering -- A Research Agenda](https://mlsanity.com/paper/2310.18648/) (2023)
- [Towards Knowledge Checking in Retrieval-augmented Generation: A Representation Perspective](https://mlsanity.com/paper/2411.14572/) (2024)
- [Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents](https://mlsanity.com/paper/2411.16740/) (2024)
- [Ask-EDA: A Design Assistant Empowered by LLM, Hybrid RAG and Abbreviation De-hallucination](https://mlsanity.com/paper/2406.06575/) (2024)
- [Towards Interpretable Radiology Report Generation via Concept Bottlenecks using a Multi-Agentic RAG](https://mlsanity.com/paper/2412.16086/) (2024)
- [Developing Retrieval Augmented Generation (RAG) based LLM Systems from PDFs: An Experience Report](https://mlsanity.com/paper/2410.15944/) (2024)
- [Know Your RAG: Dataset Taxonomy and Generation Strategies for Evaluating RAG Systems](https://mlsanity.com/paper/2411.19710/) (2024)
- [Advanced System Integration: Analyzing OpenAPI Chunking for Retrieval-Augmented Generation](https://mlsanity.com/paper/2411.19804/) (2024)
- [StackRAG Agent: Improving Developer Answers with Retrieval-Augmented Generation](https://mlsanity.com/paper/2406.13840/) (2024)
- [Path-RAG: Knowledge-Guided Key Region Retrieval for Open-ended Pathology Visual Question Answering](https://mlsanity.com/paper/2411.17073/) (2024)

## Built on by (most cited first, 3 of 3)

- [GraphRAG for Engineering Diagrams: ChatP&ID Enables LLM Interaction with P&IDs](https://mlsanity.com/paper/2603.22528/) (2026)
- [Towards AI Evaluation in Domain-Specific RAG Systems: The AgriHubi Case Study](https://mlsanity.com/paper/2602.02208/) (2026)
- [Context Before Code: An Experience Report on Vibe Coding in Practice](https://mlsanity.com/paper/2603.11073/) (2026)

## Abstract (arXiv)

Retrieval-Augmented Generation (RAG) systems are emerging as a key approach for grounding Large Language Models (LLMs) in external knowledge, addressing limitations in factual accuracy and contextual relevance. However, there is a lack of empirical studies that report on the development of RAG-based implementations grounded in real-world use cases, evaluated through general user involvement, and accompanied by systematic documentation of lessons learned. This paper presents five domain-specific RAG applications developed for real-world scenarios across governance, cybersecurity, agriculture, industrial research, and medical diagnostics. Each system incorporates multilingual OCR, semantic retrieval via vector embeddings, and domain-adapted LLMs, deployed through local servers or cloud APIs to meet distinct user needs. A web-based evaluation involving a total of 100 participants assessed the systems across six dimensions: (i) Ease of Use, (ii) Relevance, (iii) Transparency, (iv) Responsiveness, (v) Accuracy, and (vi) Likelihood of Recommendation. Based on user feedback and our development experience, we documented twelve key lessons learned, highlighting technical, operational, and ethical challenges affecting the reliability and usability of RAG systems in practice.