# LitMC-BERT: transformer-based multi-label classification of biomedical literature with an application on COVID-19 literature curation

arXiv: https://arxiv.org/abs/2204.08649 · HTML: https://mlsanity.com/paper/2204.08649/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 51 references that are in the map.
- First posted on arXiv 19 Apr 2022.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets](https://mlsanity.com/paper/1906.05474/) (2019)
- [Auxiliary Tasks in Multi-task Learning](https://mlsanity.com/paper/1805.06334/) (2018)
- [An Empirical Study on Large-Scale Multi-Label Text Classification Including Few and Zero-Shot Labels](https://mlsanity.com/paper/2010.01653/) (2020)
- [Large-scale Multi-label Text Classification - Revisiting Neural Networks](https://mlsanity.com/paper/1312.5419/) (2013)
- [ML-Net: multi-label classification of biomedical texts with deep neural networks](https://mlsanity.com/paper/1811.05475/) (2018)
- [Multi-Label Text Classification using Attention-based Graph Neural Network](https://mlsanity.com/paper/2003.11644/) (2020)
- [BioConceptVec: creating and evaluating literature-based biomedical concept embeddings on a large scale](https://mlsanity.com/paper/1912.10846/) (2019)
- [A multi-task deep learning model for the classification of Age-related Macular Degeneration](https://mlsanity.com/paper/1812.00422/) (2018)
- [Document Classification for COVID-19 Literature](https://mlsanity.com/paper/2006.13816/) (2020)
- [Artificial Intelligence (AI) in Action: Addressing the COVID-19 Pandemic with Natural Language Processing (NLP)](https://mlsanity.com/paper/2010.16413/) (2020)
- [Multi-label natural language processing to identify diagnosis and procedure codes from MIMIC-III inpatient notes](https://mlsanity.com/paper/2003.07507/) (2020)
- [Navigating the landscape of COVID-19 research through literature analysis: A bird's eye view](https://mlsanity.com/paper/2008.03397/) (2020)

## Built on by (most cited first, 6 of 6)

- [Benchmarking large language models for biomedical natural language processing applications and recommendations](https://mlsanity.com/paper/2305.16326/) (2023)
- [Multi-label classification for biomedical literature: an overview of the BioCreative VII LitCovid Track for COVID-19 literature topic annotations](https://mlsanity.com/paper/2204.09781/) (2022)
- [Large Language Models for Healthcare Text Classification: A Systematic Review](https://mlsanity.com/paper/2503.01159/) (2025)
- [Large Language Models for History, Philosophy, and Sociology of Science: Interpretive Uses, Methodological Challenges, and Critical Perspectives](https://mlsanity.com/paper/2506.12242/) (2025)
- [Automated Text Mining of Experimental Methodologies from Biomedical Literature](https://mlsanity.com/paper/2404.13779/) (2024)
- [QUAD-LLM-MLTC: Large Language Models Ensemble Learning for Healthcare Text Multi-Label Classification](https://mlsanity.com/paper/2502.14189/) (2025)

## Abstract (arXiv)

The rapid growth of biomedical literature poses a significant challenge for curation and interpretation. This has become more evident during the COVID-19 pandemic. LitCovid, a literature database of COVID-19 related papers in PubMed, has accumulated over 180,000 articles with millions of accesses. Approximately 10,000 new articles are added to LitCovid every month. A main curation task in LitCovid is topic annotation where an article is assigned with up to eight topics, e.g., Treatment and Diagnosis. The annotated topics have been widely used both in LitCovid (e.g., accounting for ~18% of total uses) and downstream studies such as network generation. However, it has been a primary curation bottleneck due to the nature of the task and the rapid literature growth. This study proposes LITMC-BERT, a transformer-based multi-label classification method in biomedical literature. It uses a shared transformer backbone for all the labels while also captures label-specific features and the correlations between label pairs. We compare LITMC-BERT with three baseline models on two datasets. Its micro-F1 and instance-based F1 are 5% and 4% higher than the current best results, respectively, and only requires ~18% of the inference time than the Binary BERT baseline. The related datasets and models are available via https://github.com/ncbi/ml-transformer.