# MedGPT: Medical Concept Prediction from Clinical Narratives

arXiv: https://arxiv.org/abs/2107.03134 · HTML: https://mlsanity.com/paper/2107.03134/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 24 references that are in the map.
- First posted on arXiv 7 Jul 2021.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction](https://mlsanity.com/paper/2005.12833/) (2020)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [ReZero is All You Need: Fast Convergence at Large Depth](https://mlsanity.com/paper/2003.04887/) (2020)
- [BEHRT: Transformer for Electronic Health Records](https://mlsanity.com/paper/1907.09538/) (2019)
- [A Diagnostic Study of Explainability Techniques for Text Classification](https://mlsanity.com/paper/2009.13295/) (2020)
- [Pre-training of Graph Augmented Transformers for Medication Recommendation](https://mlsanity.com/paper/1906.00346/) (2019)
- [Memory Transformer](https://mlsanity.com/paper/2006.11527/) (2020)
- [Multi-domain Clinical Natural Language Processing with MedCAT: the Medical Concept Annotation Toolkit](https://mlsanity.com/paper/2010.01165/) (2020)
- [Explicit Sparse Transformer: Concentrated Attention Through Explicit Selection](https://mlsanity.com/paper/1912.11637/) (2019)
- [Talking-Heads Attention](https://mlsanity.com/paper/2003.02436/) (2020)
- [RealFormer: Transformer Likes Residual Attention](https://mlsanity.com/paper/2012.11747/) (2020)
- [MedCATTrainer: A Biomedical Free Text Annotation Interface with Active Learning and Research Use Case Specific Customisation](https://mlsanity.com/paper/1907.07322/) (2019)

## Built on by (most cited first, 24 of 24)

- [A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics](https://mlsanity.com/paper/2310.05694/) (2023)
- [Pre-trained Language Models in Biomedical Domain: A Systematic Survey](https://mlsanity.com/paper/2110.05006/) (2021)
- [Recovering Private Text in Federated Learning of Language Models](https://mlsanity.com/paper/2205.08514/) (2022)
- [K2: A Foundation Language Model for Geoscience Knowledge Understanding and Utilization](https://mlsanity.com/paper/2306.05064/) (2023)
- [Event Stream GPT: A Data Pre-processing and Modeling Library for Generative, Pre-trained Transformers over Continuous-time Sequences of Complex Events](https://mlsanity.com/paper/2306.11547/) (2023)
- [OceanGPT: A Large Language Model for Ocean Science Tasks](https://mlsanity.com/paper/2310.02031/) (2023)
- [Large Language Models for Medicine: A Survey](https://mlsanity.com/paper/2405.13055/) (2024)
- [A Continued Pretrained LLM Approach for Automatic Medical Note Generation](https://mlsanity.com/paper/2403.09057/) (2024)
- [Multilingual Large Language Models: A Systematic Survey](https://mlsanity.com/paper/2411.11072/) (2024)
- [In-BoXBART: Get Instructions into Biomedical Multi-Task Learning](https://mlsanity.com/paper/2204.07600/) (2022)
- [Open Challenges and Opportunities in Federated Foundation Models Towards Biomedical Healthcare](https://mlsanity.com/paper/2405.06784/) (2024)
- [Fleming-R1: Toward Expert-Level Medical Reasoning via Reinforcement Learning](https://mlsanity.com/paper/2509.15279/) (2025)
- [Private Yet Social: How LLM Chatbots Support and Challenge Eating Disorder Recovery](https://mlsanity.com/paper/2412.11656/) (2024)
- [A Survey of Pre-trained Language Models for Processing Scientific Text](https://mlsanity.com/paper/2401.17824/) (2024)
- [PEACE: Empowering Geologic Map Holistic Understanding with MLLMs](https://mlsanity.com/paper/2501.06184/) (2025)
- [FoMoH: A clinically meaningful foundation model evaluation for structured electronic health records](https://mlsanity.com/paper/2505.16941/) (2025)
- [LabTOP: A Unified Model for Lab Test Outcome Prediction on Electronic Health Records](https://mlsanity.com/paper/2502.14259/) (2025)
- [Extensive Evaluation of Transformer-based Architectures for Adverse Drug Events Extraction](https://mlsanity.com/paper/2306.05276/) (2023)
- [OrthoDoc: Multimodal Large Language Model for Assisting Diagnosis in Computed Tomography](https://mlsanity.com/paper/2409.09052/) (2024)
- [VesSAM: Efficient Multi-Prompting for Segmenting Complex Vessel](https://mlsanity.com/paper/2511.00981/) (2025)
- [Correcting Visual Blur Induced by Attention Distraction to Reduce Hallucinations: Algorithm and Theory](https://mlsanity.com/paper/2605.24602/) (2026)
- [FinLLM-B: When Large Language Models Meet Financial Breakout Trading](https://mlsanity.com/paper/2402.07536/) (2024)
- [Detecci'on Autom'atica de Patolog'ias en Notas Cl'inicas en Espa~nol Combinando Modelos de Lenguaje y Ontolog'ias M'edicos](https://mlsanity.com/paper/2410.00616/) (2024)
- [Automatic detection of diseases in Spanish clinical notes combining medical language models and ontologies](https://mlsanity.com/paper/2412.03176/) (2024)

## Abstract (arXiv)

The data available in Electronic Health Records (EHRs) provides the opportunity to transform care, and the best way to provide better care for one patient is through learning from the data available on all other patients. Temporal modelling of a patient's medical history, which takes into account the sequence of past events, can be used to predict future events such as a diagnosis of a new disorder or complication of a previous or existing disorder. While most prediction approaches use mostly the structured data in EHRs or a subset of single-domain predictions and outcomes, we present MedGPT a novel transformer-based pipeline that uses Named Entity Recognition and Linking tools (i.e. MedCAT) to structure and organize the free text portion of EHRs and anticipate a range of future medical events (initially disorders). Since a large portion of EHR data is in text form, such an approach benefits from a granular and detailed view of a patient while introducing modest additional noise. MedGPT effectively deals with the noise and the added granularity, and achieves a precision of 0.344, 0.552 and 0.640 (vs LSTM 0.329, 0.538 and 0.633) when predicting the top 1, 3 and 5 candidate future disorders on real world hospital data from King's College Hospital, London, UK (textasciitilde600k patients). We also show that our model captures medical knowledge by testing it on an experimental medical multiple choice question answering task, and by examining the attentional focus of the model using gradient-based saliency methods.