# Scalable Medication Extraction and Discontinuation Identification from Electronic Health Records Using Large Language Models

arXiv: https://arxiv.org/abs/2506.11137 · HTML: https://mlsanity.com/paper/2506.11137/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 60 references that are in the map.
- First posted on arXiv 10 Jun 2025.

## Builds on (most cited first, 21 of 21)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Domain Adaptation for Medical Image Analysis: A Survey](https://mlsanity.com/paper/2102.09508/) (2021)
- [MEDITRON-70B: Scaling Medical Pretraining for Large Language Models](https://mlsanity.com/paper/2311.16079/) (2023)
- [Large Language Models are Few-Shot Clinical Information Extractors](https://mlsanity.com/paper/2205.12689/) (2022)
- [Benchmarking large language models for biomedical natural language processing applications and recommendations](https://mlsanity.com/paper/2305.16326/) (2023)
- [PHEE: A Dataset for Pharmacovigilance Event Extraction from Text](https://mlsanity.com/paper/2210.12560/) (2022)
- [CoMT: Chain-of-Medical-Thought Reduces Hallucination in Medical Report Generation](https://mlsanity.com/paper/2406.11451/) (2024)
- [Guiding Clinical Reasoning with Large Language Models via Knowledge Seeds](https://mlsanity.com/paper/2403.06609/) (2024)
- [Probabilistic Medical Predictions of Large Language Models](https://mlsanity.com/paper/2408.11316/) (2024)
- [METS-CoV: A Dataset of Medical Entity and Targeted Sentiment on COVID-19 Related Tweets](https://mlsanity.com/paper/2209.13773/) (2022)
- [BRIDGE: Benchmarking Large Language Models for Understanding Real-world Clinical Practice Text](https://mlsanity.com/paper/2504.19467/) (2025)
- [Large Language Models Leverage External Knowledge to Extend Clinical Insight Beyond Language Boundaries](https://mlsanity.com/paper/2305.10163/) (2023)
- [Contextualized Medication Information Extraction Using Transformer-based Deep Learning Architectures](https://mlsanity.com/paper/2303.08259/) (2023)
- [Toward Understanding Clinical Context of Medication Change Events in Clinical Narratives](https://mlsanity.com/paper/2011.08835/) (2020)
- [Extracting Medication Changes in Clinical Narratives using Pre-trained Language Models](https://mlsanity.com/paper/2208.08417/) (2022)

## Built on by (most cited first, 1 of 1)

- [DT-Transformer: A Foundation Model for Disease Trajectory Prediction on a Real-world Health System](https://mlsanity.com/paper/2605.14227/) (2026)

## Abstract (arXiv)

Identifying medication discontinuations in electronic health records (EHRs) is vital for patient safety but is often hindered by information being buried in unstructured notes. This study aims to evaluate the capabilities of advanced open-sourced and proprietary large language models (LLMs) in extracting medications and classifying their medication status from EHR notes, focusing on their scalability on medication information extraction without human annotation. We collected three EHR datasets from diverse sources to build the evaluation benchmark. We evaluated 12 advanced LLMs and explored multiple LLM prompting strategies. Performance on medication extraction, medication status classification, and their joint task (extraction then classification) was systematically compared across all experiments. We found that LLMs showed promising performance on the medication extraction and discontinuation classification from EHR notes. GPT-4o consistently achieved the highest average F1 scores in all tasks under zero-shot setting - 94.0% for medication extraction, 78.1% for discontinuation classification, and 72.7% for the joint task. Open-sourced models followed closely, Llama-3.1-70B-Instruct achieved the highest performance in medication status classification on the MIV-Med dataset (68.7%) and in the joint task on both the Re-CASI (76.2%) and MIV-Med (60.2%) datasets. Medical-specific LLMs demonstrated lower performance compared to advanced general-domain LLMs. Few-shot learning generally improved performance, while CoT reasoning showed inconsistent gains. LLMs demonstrate strong potential for medication extraction and discontinuation identification on EHR notes, with open-sourced models offering scalable alternatives to proprietary systems and few-shot can further improve LLMs' capability.