# The Shaky Foundations of Clinical Foundation Models: A Survey of Large Language Models and Foundation Models for EMRs

arXiv: https://arxiv.org/abs/2303.12961 · HTML: https://mlsanity.com/paper/2303.12961/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 97 references that are in the map.
- First posted on arXiv 22 Mar 2023.

## Builds on (most cited first, 30 of 42)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Quantifying Memorization Across Neural Language Models](https://mlsanity.com/paper/2202.07646/) (2022)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [Structure and Content-Guided Video Synthesis with Diffusion Models](https://mlsanity.com/paper/2302.03011/) (2023)
- [Memory-Based Model Editing at Scale](https://mlsanity.com/paper/2206.06520/) (2022)
- [VIMA: General Robot Manipulation with Multimodal Prompts](https://mlsanity.com/paper/2210.03094/) (2022)
- [Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction](https://mlsanity.com/paper/2005.12833/) (2020)
- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [Scalable and accurate deep learning for electronic health records](https://mlsanity.com/paper/1801.07860/) (2018)
- [The Impact of AI on Developer Productivity: Evidence from GitHub Copilot](https://mlsanity.com/paper/2302.06590/) (2023)
- [Doctor AI: Predicting Clinical Events via Recurrent Neural Networks](https://mlsanity.com/paper/1511.05942/) (2015)
- [Large Language Models are Few-Shot Clinical Information Extractors](https://mlsanity.com/paper/2205.12689/) (2022)
- [Can large language models reason about medical questions?](https://mlsanity.com/paper/2207.08143/) (2022)
- [Do Users Write More Insecure Code with AI Assistants?](https://mlsanity.com/paper/2211.03622/) (2022)
- [Towards artificial general intelligence via a multimodal foundation model](https://mlsanity.com/paper/2110.14378/) (2021)
- [ChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reports](https://mlsanity.com/paper/2212.14882/) (2022)
- [The Cost of Training NLP Models: A Concise Overview](https://mlsanity.com/paper/2004.08900/) (2020)
- [Do We Still Need Clinical Language Models?](https://mlsanity.com/paper/2302.08091/) (2023)
- [Language Models Are An Effective Patient Representation Learning Technique For Electronic Health Record Data](https://mlsanity.com/paper/2001.05295/) (2020)
- [Medically Aware GPT-3 as a Data Generator for Medical Dialogue Summarization](https://mlsanity.com/paper/2110.07356/) (2021)
- [CEHR-BERT: Incorporating temporal information from structured EHR data to improve prediction tasks](https://mlsanity.com/paper/2111.08585/) (2021)
- [Integrated multimodal artificial intelligence framework for healthcare applications](https://mlsanity.com/paper/2202.12998/) (2022)
- [Hi-BEHRT: Hierarchical Transformer-based model for accurate prediction of clinical events using multimodal longitudinal electronic health records](https://mlsanity.com/paper/2106.11360/) (2021)
- [Deep Representation Learning of Patient Data from Electronic Health Records (EHR): A Systematic Review](https://mlsanity.com/paper/2010.02809/) (2020)
- [GPT-3 Models are Poor Few-Shot Learners in the Biomedical Domain](https://mlsanity.com/paper/2109.02555/) (2021)

## Built on by (most cited first, 29 of 29)

- [EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Models](https://mlsanity.com/paper/2307.02028/) (2023)
- [Evaluation of GPT-3.5 and GPT-4 for supporting real-world information needs in healthcare delivery](https://mlsanity.com/paper/2304.13714/) (2023)
- [Data-Centric Foundation Models in Computational Healthcare: A Survey](https://mlsanity.com/paper/2401.02458/) (2024)
- [Multimodal Clinical Benchmark for Emergency Care (MC-BEC): A Comprehensive Benchmark for Evaluating Foundation Models in Emergency Medicine](https://mlsanity.com/paper/2311.04937/) (2023)
- [MedAgentGym: A Scalable Agentic Training Environment for Code-Centric Reasoning in Biomedical Data Science](https://mlsanity.com/paper/2506.04405/) (2025)
- [Knowledge-Infused Prompting: Assessing and Advancing Clinical Text Data Generation with Large Language Models](https://mlsanity.com/paper/2311.00287/) (2023)
- [Training LLMs for EHR-Based Reasoning Tasks via Reinforcement Learning](https://mlsanity.com/paper/2505.24105/) (2025)
- [Educational Personalized Learning Path Planning with Large Language Models](https://mlsanity.com/paper/2407.11773/) (2024)
- [Identifying Reasons for Contraceptive Switching from Real-World Data Using Large Language Models](https://mlsanity.com/paper/2402.03597/) (2024)
- [Cost-Sensitive Best Subset Selection for Logistic Regression: A Mixed-Integer Conic Optimization Perspective](https://mlsanity.com/paper/2310.05464/) (2023)
- [End-to-End Chart Summarization via Visual Chain-of-Thought in Vision-Language Models](https://mlsanity.com/paper/2502.17589/) (2025)
- [Interpretable and Robust Dialogue State Tracking via Natural Language Summarization with LLMs](https://mlsanity.com/paper/2503.08857/) (2025)
- [Data augmentation method for modeling health records with applications to clopidogrel treatment failure detection](https://mlsanity.com/paper/2402.18046/) (2024)
- [Embedding-based Multimodal Learning on Pan-Squamous Cell Carcinomas for Improved Survival Outcomes](https://mlsanity.com/paper/2406.08521/) (2024)
- [Large Vision-Language Models for Remote Sensing Visual Question Answering](https://mlsanity.com/paper/2411.10857/) (2024)
- [Fault Diagnosis in Power Grids with Large Language Model](https://mlsanity.com/paper/2407.08836/) (2024)
- [Enhancing Agricultural Machinery Management through Advanced LLM Integration](https://mlsanity.com/paper/2407.20588/) (2024)
- [Foundation Model or Finetune? Evaluation of few-shot semantic segmentation for river pollution](https://mlsanity.com/paper/2409.03754/) (2024)
- [Generating Medically-Informed Explanations for Depression Detection using LLMs](https://mlsanity.com/paper/2503.14671/) (2025)
- [Machine Learning Predicts Upper Secondary Education Dropout as Early as the End of Primary School](https://mlsanity.com/paper/2403.14663/) (2024)
- [LLMs for Enhanced Agricultural Meteorological Recommendations](https://mlsanity.com/paper/2408.04640/) (2024)
- [Enhancing AI-Driven Psychological Consultation: Layered Prompts with Large Language Models](https://mlsanity.com/paper/2408.16276/) (2024)
- [Indication Finding: a novel use case for representation learning](https://mlsanity.com/paper/2410.19174/) (2024)
- [Align, Generate, Learn: A Novel Closed-Loop Framework for Cross-Lingual In-Context Learning](https://mlsanity.com/paper/2412.08955/) (2024)
- [Enhancing Contrastive Demonstration Selection with Semantic Diversity for Robust In-Context Machine Translation](https://mlsanity.com/paper/2504.09305/) (2025)
- [Harnessing Generative LLMs for Enhanced Financial Event Entity Extraction Performance](https://mlsanity.com/paper/2504.14633/) (2025)
- [Revolutionizing Radiology Workflow with Factual and Efficient CXR Report Generation](https://mlsanity.com/paper/2506.01118/) (2025)
- [Epistemic Compression: The Case for Deliberate Ignorance in High-Stakes AI](https://mlsanity.com/paper/2603.25033/) (2026)
- [Preferred, Not Safer: Pairwise Preference Is a Poor Proxy for Clinical Safety](https://mlsanity.com/paper/2608.02617/) (2026)

## Abstract (arXiv)

The successes of foundation models such as ChatGPT and AlphaFold have spurred significant interest in building similar models for electronic medical records (EMRs) to improve patient care and hospital operations. However, recent hype has obscured critical gaps in our understanding of these models' capabilities. We review over 80 foundation models trained on non-imaging EMR data (i.e. clinical text and/or structured data) and create a taxonomy delineating their architectures, training data, and potential use cases. We find that most models are trained on small, narrowly-scoped clinical datasets (e.g. MIMIC-III) or broad, public biomedical corpora (e.g. PubMed) and are evaluated on tasks that do not provide meaningful insights on their usefulness to health systems. In light of these findings, we propose an improved evaluation framework for measuring the benefits of clinical foundation models that is more closely grounded to metrics that matter in healthcare.