# A Continued Pretrained LLM Approach for Automatic Medical Note Generation

arXiv: https://arxiv.org/abs/2403.09057 · HTML: https://mlsanity.com/paper/2403.09057/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 26 references that are in the map.
- First posted on arXiv 14 Mar 2024.

## Builds on (most cited first, 18 of 18)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining](https://mlsanity.com/paper/2210.10341/) (2022)
- [Orca: Progressive Learning from Complex Explanation Traces of GPT-4](https://mlsanity.com/paper/2306.02707/) (2023)
- [MedDialog: Two Large-scale Medical Dialogue Datasets](https://mlsanity.com/paper/2004.03329/) (2020)
- [PMC-LLaMA: Towards Building Open-source Language Models for Medicine](https://mlsanity.com/paper/2304.14454/) (2023)
- [Leveraging Pretrained Models for Automatic Summarization of Doctor-Patient Conversations](https://mlsanity.com/paper/2109.12174/) (2021)
- [MedGPT: Medical Concept Prediction from Clinical Narratives](https://mlsanity.com/paper/2107.03134/) (2021)
- [Continued Pretraining for Better Zero- and Few-Shot Promptability](https://mlsanity.com/paper/2210.10258/) (2022)

## Built on by (most cited first, 30 of 30)

- [Agentic Large Language Models, a survey](https://mlsanity.com/paper/2503.23037/) (2025)
- [Synthetic continued pretraining](https://mlsanity.com/paper/2409.07431/) (2024)
- [Llama-3.1-FoundationAI-SecurityLLM-Base-8B Technical Report](https://mlsanity.com/paper/2504.21039/) (2025)
- [TWIN-GPT: Digital Twins for Clinical Trials via Large Language Model](https://mlsanity.com/paper/2404.01273/) (2024)
- [Towards User-level Private Reinforcement Learning with Human Feedback](https://mlsanity.com/paper/2502.17515/) (2025)
- [A Case Study Exploring the Current Landscape of Synthetic Medical Record Generation with Commercial LLMs](https://mlsanity.com/paper/2504.14657/) (2025)
- [Domain Adaptation of Foundation LLMs for e-Commerce](https://mlsanity.com/paper/2501.09706/) (2025)
- [RoQLlama: A Lightweight Romanian Adapted Language Model](https://mlsanity.com/paper/2410.04269/) (2024)
- [MedTVT-R1: A Multimodal LLM Empowering Medical Reasoning and Diagnosis](https://mlsanity.com/paper/2506.18512/) (2025)
- [RedOne: Revealing Domain-specific LLM Post-Training in Social Networking Services](https://mlsanity.com/paper/2507.10605/) (2025)
- [EHRStruct: A Comprehensive Benchmark Framework for Evaluating Large Language Models on Structured Electronic Health Record Tasks](https://mlsanity.com/paper/2511.08206/) (2025)
- [Domain-Specific Pretraining of Language Models: A Comparative Study in the Medical Field](https://mlsanity.com/paper/2407.14076/) (2024)
- [Knowledge Hierarchy Guided Biological-Medical Dataset Distillation for Domain LLM Training](https://mlsanity.com/paper/2501.15108/) (2025)
- [Enhancing Domain-Specific Encoder Models with LLM-Generated Data: How to Leverage Ontologies, and How to Do Without Them](https://mlsanity.com/paper/2503.22006/) (2025)
- [OrthoDoc: Multimodal Large Language Model for Assisting Diagnosis in Computed Tomography](https://mlsanity.com/paper/2409.09052/) (2024)
- [TN-Eval: Rubric and Evaluation Protocols for Measuring the Quality of Behavioral Therapy Notes](https://mlsanity.com/paper/2503.20648/) (2025)
- [Watermarking Large Language Models and the Generated Content: Opportunities and Challenges](https://mlsanity.com/paper/2410.19096/) (2024)
- [GeneSUM: Large Language Model-based Gene Summary Extraction](https://mlsanity.com/paper/2412.18154/) (2024)
- [Exploring LLM-based Data Annotation Strategies for Medical Dialogue Preference Alignment](https://mlsanity.com/paper/2410.04112/) (2024)
- [ALKAFI-LLAMA3: Fine-Tuning LLMs for Precise Legal Understanding in Palestine](https://mlsanity.com/paper/2412.14771/) (2024)
- [Enhancing Hepatopathy Clinical Trial Efficiency: A Secure, Large Language Model-Powered Pre-Screening Pipeline](https://mlsanity.com/paper/2502.18531/) (2025)
- [Knowledge-Driven Agentic Scientific Corpus Distillation Framework for Biomedical Large Language Models Training](https://mlsanity.com/paper/2504.19565/) (2025)
- [Enhancing Clinical Note Generation with ICD-10, Clinical Ontology Knowledge Graphs, and Chain-of-Thought Prompting Using GPT-4](https://mlsanity.com/paper/2512.05256/) (2025)
- [WaterMoE: Expert-Routing-based Watermarking for High Fidelity and Efficiency](https://mlsanity.com/paper/2607.13099/) (2026)
- [Searching for Best Practices in Medical Transcription with Large Language Model](https://mlsanity.com/paper/2410.03797/) (2024)
- [Medalyze: Lightweight Medical Report Summarization Application Using FLAN-T5-Large](https://mlsanity.com/paper/2505.17059/) (2025)
- [ChiMed 2.0: Advancing Chinese Medical Dataset in Facilitating Large Language Modeling](https://mlsanity.com/paper/2507.15275/) (2025)
- [FedSODA: Federated Fine-tuning of LLMs via Similarity Group Pruning and Orchestrated Distillation Alignment](https://mlsanity.com/paper/2508.12727/) (2025)
- [MM-Telco: Benchmarks and Multimodal Large Language Models for Telecom Applications](https://mlsanity.com/paper/2511.13131/) (2025)
- [Continually self-improving AI](https://mlsanity.com/paper/2603.18073/) (2026)

## Abstract (arXiv)

LLMs are revolutionizing NLP tasks. However, the use of the most advanced LLMs, such as GPT-4, is often prohibitively expensive for most specialized fields. We introduce HEAL, the first continuously trained 13B LLaMA2-based LLM that is purpose-built for medical conversations and measured on automated scribing. Our results demonstrate that HEAL outperforms GPT-4 and PMC-LLaMA in PubMedQA, with an accuracy of 78.4%. It also achieves parity with GPT-4 in generating medical notes. Remarkably, HEAL surpasses GPT-4 and Med-PaLM 2 in identifying more correct medical concepts and exceeds the performance of human scribes and other comparable models in correctness and completeness.