# Development and Testing of Retrieval Augmented Generation in Large Language Models -- A Case Study Report

arXiv: https://arxiv.org/abs/2402.01733 · HTML: https://mlsanity.com/paper/2402.01733/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Medical and biomedical RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 4 of its 32 references that are in the map.
- First posted on arXiv 29 Jan 2024.

## Placed in

- Retrieval-augmented generation / [Medical and biomedical RAG](https://mlsanity.com/rag/medical-and-biomedical-rag/)

## Builds on (most cited first, 4 of 4)

- [Almanac: Retrieval-Augmented Language Models for Clinical Medicine](https://mlsanity.com/paper/2303.01229/) (2023)
- [The role of artificial intelligence in achieving the Sustainable Development Goals](https://mlsanity.com/paper/1905.00501/) (2019)
- [Integrating UMLS Knowledge into Large Language Models for Medical Question Answering](https://mlsanity.com/paper/2310.02778/) (2023)
- [A Survey on AI Sustainability: Emerging Trends on Learning Algorithms and Research Challenges](https://mlsanity.com/paper/2205.03824/) (2022)

## Built on by (most cited first, 13 of 13)

- [Can Large Language Models Match the Conclusions of Systematic Reviews?](https://mlsanity.com/paper/2505.22787/) (2025)
- [RAGVA: Engineering Retrieval Augmented Generation-based Virtual Assistants in Practice](https://mlsanity.com/paper/2502.14930/) (2025)
- [Generative Artificial Intelligence: Evolving Technology, Growing Societal Impact, and Opportunities for Information Systems Research](https://mlsanity.com/paper/2503.05770/) (2025)
- [AlzheimerRAG: Multimodal Retrieval Augmented Generation for Clinical Use Cases using PubMed articles](https://mlsanity.com/paper/2412.16701/) (2024)
- [Multi-hop Reasoning and Retrieval in Embedding Space: Leveraging Large Language Models with Knowledge](https://mlsanity.com/paper/2603.13266/) (2026)
- [Lightweight Large Language Model for Medication Enquiry: Med-Pal](https://mlsanity.com/paper/2407.12822/) (2024)
- [Enhancing Large Language Models with Domain-Specific Knowledge: The Case in Topological Materials](https://mlsanity.com/paper/2409.13732/) (2024)
- [Improving TCM Question Answering through Tree-Organized Self-Reflective Retrieval with LLMs](https://mlsanity.com/paper/2502.09156/) (2025)
- [FHIR-RAG-MEDS: Integrating HL7 FHIR with Retrieval-Augmented Large Language Models for Enhanced Medical Decision Support](https://mlsanity.com/paper/2509.07706/) (2025)
- [Hybrid Student-Teacher Large Language Model Refinement for Cancer Toxicity Symptom Extraction](https://mlsanity.com/paper/2408.04775/) (2024)
- [RAG-based Architectures for Drug Side Effect Retrieval in LLMs](https://mlsanity.com/paper/2507.13822/) (2025)
- [MedAlign: A Synergistic Framework of Multimodal Preference Optimization and Federated Meta-Cognitive Reasoning](https://mlsanity.com/paper/2510.21093/) (2025)
- [Mapping the RAG Landscape: A Four Axis Taxonomy of Efficiency, Defense, Interactivity, and Reasoning](https://mlsanity.com/paper/2610.01936/) (2026)

## Abstract (arXiv)

Purpose: Large Language Models (LLMs) hold significant promise for medical applications. Retrieval Augmented Generation (RAG) emerges as a promising approach for customizing domain knowledge in LLMs. This case study presents the development and evaluation of an LLM-RAG pipeline tailored for healthcare, focusing specifically on preoperative medicine. Methods: We developed an LLM-RAG model using 35 preoperative guidelines and tested it against human-generated responses, with a total of 1260 responses evaluated. The RAG process involved converting clinical documents into text using Python-based frameworks like LangChain and Llamaindex, and processing these texts into chunks for embedding and retrieval. Vector storage techniques and selected embedding models to optimize data retrieval, using Pinecone for vector storage with a dimensionality of 1536 and cosine similarity for loss metrics. Human-generated answers, provided by junior doctors, were used as a comparison. Results: The LLM-RAG model generated answers within an average of 15-20 seconds, significantly faster than the 10 minutes typically required by humans. Among the basic LLMs, GPT4.0 exhibited the best accuracy of 80.1%. This accuracy was further increased to 91.4% when the model was enhanced with RAG. Compared to the human-generated instructions, which had an accuracy of 86.3%, the performance of the GPT4.0 RAG model demonstrated non-inferiority (p=0.610). Conclusions: In this case study, we demonstrated a LLM-RAG model for healthcare implementation. The pipeline shows the advantages of grounded knowledge, upgradability, and scalability as important aspects of healthcare LLM deployment.