mlsanityData to 7 Oct 2026

Almanac: Retrieval-Augmented Language Models for Clinical Medicine

arXiv 2303.01229 ↗First posted 1 Mar 2023cs.CL cs.AIcited by 122

On the map: Retrieval-augmented generation → Medical and biomedical RAG

Abstract (arXiv)

Large-language models have recently demonstrated impressive zero-shot capabilities in a variety of natural language tasks such as summarization, dialogue generation, and question-answering. Despite many promising applications in clinical medicine, adoption of these models in real-world settings has been largely limited by their tendency to generate incorrect and sometimes even toxic statements. In this study, we develop Almanac, a large language model framework augmented with retrieval capabilities for medical guideline and treatment recommendations. Performance on a novel dataset of clinical scenarios (n = 130) evaluated by a panel of 5 board-certified and resident physicians demonstrates significant increases in factuality (mean of 18% at p-value < 0.05) across all specialties, with improvements in completeness and safety. Our results demonstrate the potential for large language models to be effective tools in the clinical decision-making process, while also emphasizing the importance of careful testing and deployment to mitigate their shortcomings.

Key facts. Cited by 122 papers in the map (data through 7 Oct 2026). Placed in Medical and biomedical RAG, one of 15 directions in Retrieval-augmented generation. Builds on 37 of its 48 references that are in the map. First posted on arXiv 1 Mar 2023.

Builds on

37 of 48 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  10. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  11. 2022Emergent Abilities of Large Language Modelscited by 2,483
  12. 2020Extracting Training Data from Large Language Modelscited by 2,144
25 more
  1. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  2. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  3. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  4. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  5. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  6. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  7. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  8. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  9. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  10. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  11. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  12. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 853
  13. 2022Galactica: A Large Language Model for Sciencecited by 736
  14. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  15. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
  16. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  17. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 294
  18. 2022Can large language models reason about medical questions?cited by 194
  19. 2021The Curious Case of Hallucinations in Neural Machine Translationcited by 161
  20. 2022Deep Bidirectional Language-Knowledge Graph Pretrainingcited by 156
  21. 2023Do We Still Need Clinical Language Models?cited by 97
  22. 2021Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuningcited by 82
  23. 2020BioMegatron: Larger Biomedical Domain Language Modelcited by 76
  24. 2020DARE: Data Augmented Relation Extraction with GPT-2cited by 43
  25. 2018What are the biases in my word embedding?cited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 14; 2024: 44; 2025: 38; 2026: 26.

122 papers.

Sort:

By area:

  1. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
  2. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 326
  3. 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 311
  4. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 215
  5. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  6. 2023PaperQA: Retrieval-Augmented Generative Agent for Scientific Researchcited by 143
  7. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  8. 2023Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Healthcited by 118
  9. 2024Towards Building Multilingual Language Model for Medicinecited by 91
  10. 2024RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generationcited by 59
  11. 2024Improving Retrieval-Augmented Generation in Medicine with Iterative Follow-up Questionscited by 54
  12. 2024DALK: Dynamic Co-Augmentation of LLMs and KG to answer Alzheimer's Disease Questions with Scientific Literaturecited by 50
38 more
  1. 2024MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answeringcited by 43
  2. 2024Rationale-Guided Retrieval Augmented Generation for Medical Question Answeringcited by 41
  3. 2023CPLLM: Clinical Prediction with Large Language Modelscited by 34
  4. 2024A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directionscited by 34
  5. 2024MEDCO: Medical Education Copilots Based on A Multi-Agent Frameworkcited by 33
  6. 2024JMLR: Joint Medical LLM and Retrieval Training for Enhancing Reasoning and Professional Question Answering Capabilitycited by 32
  7. 2024BlendFilter: Advancing Retrieval-Augmented Large Language Models via Query Generation Blending and Knowledge Filteringcited by 29
  8. 2024TC-RAG:Turing-Complete RAG's Case study on Medical LLM Systemscited by 28
  9. 2024MEDIC: Comprehensive Evaluation of Leading Indicators for LLM Safety and Utility in Clinical Applicationscited by 25
  10. 2023MKRAG: Medical Knowledge Retrieval Augmented Generation for Medical Question Answeringcited by 23
  11. 2023Large language models in medicine: the potentials and pitfallscited by 21
  12. 2024RAM-EHR: Retrieval Augmentation Meets Clinical Predictions on Electronic Health Recordscited by 21
  13. 2024Improving Scientific Hypothesis Generation with Knowledge Grounded Large Language Modelscited by 21
  14. 2024PaCE: Parsimonious Concept Engineering for Large Language Modelscited by 19
  15. 2024GP-VLS: A general-purpose vision language model for surgerycited by 16
  16. 2023An Automatic Evaluation Framework for Multi-turn Medical Consultations Capabilities of Large Language Modelscited by 13
  17. 2024Development and Testing of Retrieval Augmented Generation in Large Language Models -- A Case Study Reportcited by 13
  18. 2024LLMs in Biomedicine: A study on clinical Named Entity Recognitioncited by 12
  19. 2024Large Language Models in the Clinic: A Comprehensive Benchmarkcited by 12
  20. 2026MedSynapse-V: Bridging Visual Perception and Clinical Intuition via Latent Memory Evolutioncited by 12
  21. 2024Enhancing Large Language Models for Clinical Decision Support by Incorporating Clinical Practice Guidelinescited by 11
  22. 2025BRIDGE: Benchmarking Large Language Models for Understanding Real-world Clinical Practice Textcited by 11
  23. 2025MedCite: Can Language Models Generate Verifiable Text for Medicine?cited by 11
  24. 2023Integrating UMLS Knowledge into Large Language Models for Medical Question Answeringcited by 10
  25. 2024RadioRAG: Online Retrieval-augmented Generation for Radiology Question Answeringcited by 10
  26. 2024Almanac Copilot: Towards Autonomous Electronic Health Record Navigationcited by 9
  27. 2024SEER: Self-Aligned Evidence Extraction for Retrieval-Augmented Generationcited by 9
  28. 2023A Generalizable Deep Learning System for Cardiac MRIcited by 8
  29. 2024From Text to Multimodality: Exploring the Evolution and Impact of Large Language Models in Medical Practicecited by 8
  30. 2024A Perspective for Adapting Generalist AI to Specialized Medical AI Applications and Their Challengescited by 8
  31. 2025Emerging Cyber Attack Risks of Medical AI Agentscited by 8
  32. 2024MedInsight: A Multi-Source Context Augmentation Framework for Generating Patient-Centric Medical Responses using Large Language Modelscited by 7
  33. 2025Multi-step retrieval and reasoning improves radiology question answering with large language modelscited by 7
  34. 2025Grounding Large Language Models in Clinical Evidence: A Retrieval-Augmented Generation System for Querying UK NICE Clinical Guidelinescited by 7
  35. 2024Bailicai: A Domain-Optimized Retrieval-Augmented Generation Framework for Medical Applicationscited by 6
  36. 2025Exploring Efficiency Frontiers of Thinking Budget in Medical Reasoning: Scaling Laws between Computational Resources and Reasoning Qualitycited by 6
  37. 2025Toward Faithful Retrieval-Augmented Generation with Sparse Autoencoderscited by 6
  38. 2024HiQA: A Hierarchical Contextual Augmentation RAG for Multi-Documents QAcited by 5

50 most cited of 122 All 122 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating Large Language Models for Evidence-Based Clinical Question Answering
  2. 2025Large Language Models with Temporal Reasoning for Longitudinal Clinical Summarization and Prediction
  3. 2023ClinicalGPT: Large Language Models Finetuned with Diverse Medical Data and Comprehensive Evaluation
  4. 2023Clinfo.ai: An Open-Source Retrieval-Augmented Large Language Model System for Answering Medical Questions using Scientific Literature
  5. 2025Retrieval-Augmented Generation in Medicine: A Scoping Review of Technical Implementations, Clinical Applications, and Ethical Considerations
  6. 2026A corpus-specific clinical RAG system matches or outperforms newer frontier LLMs on HealthBench
  7. 2025MedGUIDE: Benchmarking Clinical Decision-Making in Large Language Models
  8. 2024Enhancing Large Language Models with Domain-specific Retrieval Augment Generation: A Case Study on Long-form Consumer Health Question Answering in Ophthalmology