# MT-Clinical BERT: Scaling Clinical Information Extraction with Multitask Learning

arXiv: https://arxiv.org/abs/2004.10220 · HTML: https://mlsanity.com/paper/2004.10220/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Apr 2020.

## Built on by (most cited first, 17 of 17)

- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [Neural Natural Language Processing for Unstructured Data in Electronic Health Records: a Review](https://mlsanity.com/paper/2107.02975/) (2021)
- [AMMU : A Survey of Transformer-based Biomedical Pretrained Language Models](https://mlsanity.com/paper/2105.00827/) (2021)
- [Natural Language Processing in Electronic Health Records in Relation to Healthcare Decision-making: A Systematic Review](https://mlsanity.com/paper/2306.12834/) (2023)
- [Can Large Language Models abstract Medical Coded Language?](https://mlsanity.com/paper/2403.10822/) (2024)
- [Pre-training Language Model Incorporating Domain-specific Heterogeneous Knowledge into A Unified Representation](https://mlsanity.com/paper/2109.01048/) (2021)
- [RAMIE: Retrieval-Augmented Multi-task Information Extraction with Large Language Models on Dietary Supplements](https://mlsanity.com/paper/2411.15700/) (2024)
- [CLIN-X: pre-trained language models and a study on cross-task transfer for concept extraction in the clinical domain](https://mlsanity.com/paper/2112.08754/) (2021)
- [Patient Outcome and Zero-shot Diagnosis Prediction with Hypernetwork-guided Multitask Learning](https://mlsanity.com/paper/2109.03062/) (2021)
- [Neural Medication Extraction: A Comparison of Recent Models in Supervised and Semi-supervised Learning Settings](https://mlsanity.com/paper/2110.10213/) (2021)
- [SuMe: A Dataset Towards Summarizing Biomedical Mechanisms](https://mlsanity.com/paper/2205.04652/) (2022)
- [Literature Based Discovery (LBD): Towards Hypothesis Generation and Knowledge Discovery in Biomedical Text Mining](https://mlsanity.com/paper/2310.03766/) (2023)
- [Building A Knowledge Graph to Enrich ChatGPT Responses in Manufacturing Service Discovery](https://mlsanity.com/paper/2404.06571/) (2024)
- [MedPT: A Massive Medical Question Answering Dataset for Brazilian-Portuguese Speakers](https://mlsanity.com/paper/2511.11878/) (2025)
- [CTBench: A Comprehensive Benchmark for Evaluating Language Model Capabilities in Clinical Trial Design](https://mlsanity.com/paper/2406.17888/) (2024)
- [Investigating Deep-Learning NLP for Automating the Extraction of Oncology Efficacy Endpoints from Scientific Literature](https://mlsanity.com/paper/2311.04925/) (2023)
- [A Question Answering Based Pipeline for Comprehensive Chinese EHR Information Extraction](https://mlsanity.com/paper/2402.11177/) (2024)

## Abstract (arXiv)

Clinical notes contain an abundance of important but not-readily accessible information about patients. Systems to automatically extract this information rely on large amounts of training data for which their exists limited resources to create. Furthermore, they are developed dis-jointly; meaning that no information can be shared amongst task-specific systems. This bottle-neck unnecessarily complicates practical application, reduces the performance capabilities of each individual solution and associates the engineering debt of managing multiple information extraction systems. We address these challenges by developing Multitask-Clinical BERT: a single deep learning model that simultaneously performs eight clinical tasks spanning entity extraction, PHI identification, language entailment and similarity by sharing representations amongst tasks. We find our single system performs competitively with all state-the-art task-specific systems while also benefiting from massive computational benefits at inference.