# Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notes

arXiv: https://arxiv.org/abs/2309.00237 · HTML: https://mlsanity.com/paper/2309.00237/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 56 references that are in the map.
- First posted on arXiv 1 Sep 2023.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data](https://mlsanity.com/paper/2304.08247/) (2023)
- [GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Records](https://mlsanity.com/paper/2203.03540/) (2022)
- [Large Language Models are Few-Shot Clinical Information Extractors](https://mlsanity.com/paper/2205.12689/) (2022)
- [Can large language models reason about medical questions?](https://mlsanity.com/paper/2207.08143/) (2022)
- [DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4](https://mlsanity.com/paper/2303.11032/) (2023)
- [Do We Still Need Clinical Language Models?](https://mlsanity.com/paper/2302.08091/) (2023)
- [A Comparative Study of Pretrained Language Models for Long Clinical Text](https://mlsanity.com/paper/2301.11847/) (2023)
- [Evaluation of GPT-3.5 and GPT-4 for supporting real-world information needs in healthcare delivery](https://mlsanity.com/paper/2304.13714/) (2023)
- [Towards Automatic Generation of Shareable Synthetic Clinical Notes Using Neural Language Models](https://mlsanity.com/paper/1905.07002/) (2019)
- [Learning to Ask Like a Physician](https://mlsanity.com/paper/2206.02696/) (2022)

## Built on by (most cited first, 30 of 51)

- [Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions](https://mlsanity.com/paper/2402.18060/) (2024)
- [Health-LLM: Large Language Models for Health Prediction via Wearable Sensor Data](https://mlsanity.com/paper/2401.06866/) (2024)
- [MedExQA: Medical Question Answering Benchmark with Multiple Explanations](https://mlsanity.com/paper/2406.06331/) (2024)
- [EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summaries](https://mlsanity.com/paper/2402.16040/) (2024)
- [JMLR: Joint Medical LLM and Retrieval Training for Enhancing Reasoning and Professional Question Answering Capability](https://mlsanity.com/paper/2402.17887/) (2024)
- [MedFuzz: Exploring the Robustness of Large Language Models in Medical Question Answering](https://mlsanity.com/paper/2406.06573/) (2024)
- [BioInstruct: Instruction Tuning of Large Language Models for Biomedical Natural Language Processing](https://mlsanity.com/paper/2310.19975/) (2023)
- [Open (Clinical) LLMs are Sensitive to Instruction Phrasings](https://mlsanity.com/paper/2407.09429/) (2024)
- [ClinicalMamba: A Generative Clinical Language Model on Longitudinal Clinical Notes](https://mlsanity.com/paper/2403.05795/) (2024)
- [Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safety](https://mlsanity.com/paper/2505.06843/) (2025)
- [A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignment](https://mlsanity.com/paper/2505.10717/) (2025)
- [Do Large Language Models Align with Core Mental Health Counseling Competencies?](https://mlsanity.com/paper/2410.22446/) (2024)
- [A Survey on Large Language Models from General Purpose to Medical Applications: Datasets, Methodologies, and Evaluations](https://mlsanity.com/paper/2406.10303/) (2024)
- [Synthetic Data Generation with LLM for Improved Depression Prediction](https://mlsanity.com/paper/2411.17672/) (2024)
- [Towards Adapting Open-Source Large Language Models for Expert-Level Clinical Note Generation](https://mlsanity.com/paper/2405.00715/) (2024)
- [CliBench: A Multifaceted and Multigranular Evaluation of Large Language Models for Clinical Decision Making](https://mlsanity.com/paper/2406.09923/) (2024)
- [BioMamba: Domain-Adaptive Biomedical Language Models](https://mlsanity.com/paper/2408.02600/) (2024)
- [DPPA: Pruning Method for Large Language Model to Model Merging](https://mlsanity.com/paper/2403.02799/) (2024)
- [CLIMB: A Benchmark of Clinical Bias in Large Language Models](https://mlsanity.com/paper/2407.05250/) (2024)
- [WisPerMed at "Discharge Me!": Advancing Text Generation in Healthcare with Large Language Models, Dynamic Expert Selection, and Priming Techniques on MIMIC-IV](https://mlsanity.com/paper/2405.11255/) (2024)
- [Lossless Compression of Large Language Model-Generated Text via Next-Token Prediction](https://mlsanity.com/paper/2505.06297/) (2025)
- [SNOBERT: A Benchmark for clinical notes entity linking in the SNOMED CT clinical terminology](https://mlsanity.com/paper/2405.16115/) (2024)
- [Entity Decomposition with Filtering: A Zero-Shot Clinical Named Entity Recognition Framework](https://mlsanity.com/paper/2407.04629/) (2024)
- [Impact of Large Language Model Assistance on Patients Reading Clinical Notes: A Mixed-Methods Study](https://mlsanity.com/paper/2401.09637/) (2024)
- [Synth-SBDH: A Synthetic Dataset of Social and Behavioral Determinants of Health for Clinical Text](https://mlsanity.com/paper/2406.06056/) (2024)
- [Disease Entity Recognition and Normalization is Improved with Large Language Model Derived Synthetic Normalized Mentions](https://mlsanity.com/paper/2410.07951/) (2024)
- [A Scoping Review of Synthetic Data Generation by Language Models in Biomedical Research and Application: Data Utility and Quality Perspectives](https://mlsanity.com/paper/2506.16594/) (2025)
- [SoK: Privacy-aware LLM in Healthcare: Threat Model, Privacy Techniques, Challenges and Recommendations](https://mlsanity.com/paper/2601.10004/) (2026)
- [Faithfulness vs. Safety: Evaluating LLM Behavior Under Counterfactual Medical Evidence](https://mlsanity.com/paper/2601.11886/) (2026)
- [Detecting the Clinical Features of Difficult-to-Treat Depression using Synthetic Data from Large Language Models](https://mlsanity.com/paper/2402.07645/) (2024)

## Abstract (arXiv)

The development of large language models tailored for handling patients' clinical notes is often hindered by the limited accessibility and usability of these notes due to strict privacy regulations. To address these challenges, we first create synthetic large-scale clinical notes using publicly available case reports extracted from biomedical literature. We then use these synthetic notes to train our specialized clinical large language model, Asclepius. While Asclepius is trained on synthetic data, we assess its potential performance in real-world applications by evaluating it using real clinical notes. We benchmark Asclepius against several other large language models, including GPT-3.5-turbo and other open-source alternatives. To further validate our approach using synthetic notes, we also compare Asclepius with its variants trained on real clinical notes. Our findings convincingly demonstrate that synthetic clinical notes can serve as viable substitutes for real ones when constructing high-performing clinical language models. This conclusion is supported by detailed evaluations conducted by both GPT-4 and medical professionals. All resources including weights, codes, and data used in the development of Asclepius are made publicly accessible for future research. (https://github.com/starmpcc/Asclepius)