# oRetrieval Augmented Generation for 10 Large Language Models and its Generalizability in Assessing Medical Fitness

arXiv: https://arxiv.org/abs/2410.08431 · HTML: https://mlsanity.com/paper/2410.08431/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Medical and biomedical RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 5 of its 28 references that are in the map.
- First posted on arXiv 11 Oct 2024.

## Placed in

- Retrieval-augmented generation / [Medical and biomedical RAG](https://mlsanity.com/rag/medical-and-biomedical-rag/)

## Builds on (most cited first, 5 of 5)

- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Almanac: Retrieval-Augmented Language Models for Clinical Medicine](https://mlsanity.com/paper/2303.01229/) (2023)
- [The role of artificial intelligence in achieving the Sustainable Development Goals](https://mlsanity.com/paper/1905.00501/) (2019)
- [Integrating UMLS Knowledge into Large Language Models for Medical Question Answering](https://mlsanity.com/paper/2310.02778/) (2023)
- [A Survey on AI Sustainability: Emerging Trends on Learning Algorithms and Research Challenges](https://mlsanity.com/paper/2205.03824/) (2022)

## Built on by (most cited first, 1 of 1)

- [GuidelineGuard: An Agentic Framework for Medical Note Evaluation with Guideline Adherence](https://mlsanity.com/paper/2411.06264/) (2024)

## Abstract (arXiv)

Large Language Models (LLMs) show potential for medical applications but often lack specialized clinical knowledge. Retrieval Augmented Generation (RAG) allows customization with domain-specific information, making it suitable for healthcare. This study evaluates the accuracy, consistency, and safety of RAG models in determining fitness for surgery and providing preoperative instructions. We developed LLM-RAG models using 35 local and 23 international preoperative guidelines and tested them against human-generated responses. A total of 3,682 responses were evaluated. Clinical documents were processed using Llamaindex, and 10 LLMs, including GPT3.5, GPT4, and Claude-3, were assessed. Fourteen clinical scenarios were analyzed, focusing on seven aspects of preoperative instructions. Established guidelines and expert judgment were used to determine correct responses, with human-generated answers serving as comparisons. The LLM-RAG models generated responses within 20 seconds, significantly faster than clinicians (10 minutes). The GPT4 LLM-RAG model achieved the highest accuracy (96.4% vs. 86.6%, p=0.016), with no hallucinations and producing correct instructions comparable to clinicians. Results were consistent across both local and international guidelines. This study demonstrates the potential of LLM-RAG models for preoperative healthcare tasks, highlighting their efficiency, scalability, and reliability.