# An Empirical Study of Multi-Task Learning on BERT for Biomedical Text Mining

arXiv: https://arxiv.org/abs/2005.02799 · HTML: https://mlsanity.com/paper/2005.02799/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 May 2020.

## Built on by (most cited first, 30 of 35)

- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [A Review of Large Language Models and Autonomous Agents in Chemistry](https://mlsanity.com/paper/2407.01603/) (2024)
- [BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model](https://mlsanity.com/paper/2204.03905/) (2022)
- [Scientific Large Language Models: A Survey on Biological & Chemical Domains](https://mlsanity.com/paper/2401.14656/) (2024)
- [AMMU : A Survey of Transformer-based Biomedical Pretrained Language Models](https://mlsanity.com/paper/2105.00827/) (2021)
- [When to Use Multi-Task Learning vs Intermediate Fine-Tuning for Pre-Trained Encoder Transfer Learning](https://mlsanity.com/paper/2205.08124/) (2022)
- [A Large-Language Model Framework for Relative Timeline Extraction from PubMed Case Reports](https://mlsanity.com/paper/2504.12350/) (2025)
- [Biomedical Language Models are Robust to Sub-optimal Tokenization](https://mlsanity.com/paper/2306.17649/) (2023)
- [Better Few-Shot Relation Extraction with Label Prompt Dropout](https://mlsanity.com/paper/2210.13733/) (2022)
- [BioREx: Improving Biomedical Relation Extraction by Leveraging Heterogeneous Datasets](https://mlsanity.com/paper/2306.11189/) (2023)
- [How Do Your Biomedical Named Entity Recognition Models Generalize to Novel Entities?](https://mlsanity.com/paper/2101.00160/) (2021)
- [Collaborative Synthesis of Patient Records through Multi-Visit Health State Inference](https://mlsanity.com/paper/2312.14646/) (2023)
- [Transformer-based Multi-task Learning for Disaster Tweet Categorisation](https://mlsanity.com/paper/2110.08010/) (2021)
- [When Does Aggregating Multiple Skills with Multi-Task Learning Work? A Case Study in Financial NLP](https://mlsanity.com/paper/2305.14007/) (2023)
- [Forecasting Clinical Risk from Textual Time Series: Structuring Narratives for Temporal AI in Healthcare](https://mlsanity.com/paper/2504.10340/) (2025)
- [Identifying Task Groupings for Multi-Task Learning Using Pointwise V-Usable Information](https://mlsanity.com/paper/2410.12774/) (2024)
- [Generative Design of Functional Metal Complexes Utilizing the Internal Knowledge of Large Language Models](https://mlsanity.com/paper/2410.18136/) (2024)
- [STG-MTL: Scalable Task Grouping for Multi-Task Learning Using Data Map](https://mlsanity.com/paper/2307.03374/) (2023)
- [UniDCP: Unifying Multiple Medical Vision-language Tasks via Dynamic Cross-modal Learnable Prompts](https://mlsanity.com/paper/2312.11171/) (2023)
- [PMOA-TTS: Introducing the PubMed Open Access Textual Times Series Corpus](https://mlsanity.com/paper/2505.20323/) (2025)
- [Efficient Extraction of Pathologies from C-Spine Radiology Reports using Multi-Task Learning](https://mlsanity.com/paper/2204.04544/) (2022)
- [Do Language Models Understand Measurements?](https://mlsanity.com/paper/2210.12694/) (2022)
- [A Cross-institutional Evaluation on Breast Cancer Phenotyping NLP Algorithms on Electronic Health Records](https://mlsanity.com/paper/2303.08448/) (2023)
- [Retrieval-Augmented Framework for LLM-Based Clinical Decision Support](https://mlsanity.com/paper/2510.01363/) (2025)
- [HARMON-E: Hierarchical Agentic Reasoning for Multimodal Oncology Notes to Extract Structured Data](https://mlsanity.com/paper/2512.19864/) (2025)
- [Explaining the Effectiveness of Multi-Task Learning for Efficient Knowledge Extraction from Spine MRI Reports](https://mlsanity.com/paper/2205.02979/) (2022)
- [Bent & Broken Bicycles: Leveraging synthetic data for damaged object re-identification](https://mlsanity.com/paper/2304.07883/) (2023)
- [Into the Single Cell Multiverse: an End-to-End Dataset for Procedural Knowledge Extraction in Biomedical Texts](https://mlsanity.com/paper/2309.01812/) (2023)
- [Coarse-to-Fine Personalized LLM Impressions for Streamlined Radiology Reports](https://mlsanity.com/paper/2508.15845/) (2025)
- [An Open-Source Dataset and A Multi-Task Model for Malay Named Entity Recognition](https://mlsanity.com/paper/2109.01293/) (2021)

## Abstract (arXiv)

Multi-task learning (MTL) has achieved remarkable success in natural language processing applications. In this work, we study a multi-task learning model with multiple decoders on varieties of biomedical and clinical natural language processing tasks such as text similarity, relation extraction, named entity recognition, and text inference. Our empirical results demonstrate that the MTL fine-tuned models outperform state-of-the-art transformer models (e.g., BERT and its variants) by 2.0% and 1.3% in biomedical and clinical domains, respectively. Pairwise MTL further demonstrates more details about which tasks can improve or decrease others. This is particularly helpful in the context that researchers are in the hassle of choosing a suitable model for new problems. The code and models are publicly available at https://github.com/ncbi-nlp/bluebert