# Large-scale investigation of weakly-supervised deep learning for the fine-grained semantic indexing of biomedical literature

arXiv: https://arxiv.org/abs/2301.09350 · HTML: https://mlsanity.com/paper/2301.09350/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 29 references that are in the map.
- First posted on arXiv 23 Jan 2023.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping](https://mlsanity.com/paper/2002.06305/) (2020)
- [Using Pre-Training Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1901.09960/) (2019)
- [Snorkel: Rapid Training Data Creation with Weak Supervision](https://mlsanity.com/paper/1711.10160/) (2017)
- [Data Programming: Creating Large Training Sets, Quickly](https://mlsanity.com/paper/1605.07723/) (2016)
- [Distribution-Balanced Loss for Multi-Label Classification in Long-Tailed Datasets](https://mlsanity.com/paper/2007.09654/) (2020)
- [Balancing Methods for Multi-label Text Classification with Long-Tailed Class Distribution](https://mlsanity.com/paper/2109.04712/) (2021)
- [Memorisation versus Generalisation in Pre-trained Language Models](https://mlsanity.com/paper/2105.00828/) (2021)
- [Coarse2Fine: Fine-grained Text Classification on Coarsely-grained Annotated Data](https://mlsanity.com/paper/2109.10856/) (2021)
- [Overview of BioASQ 2022: The tenth BioASQ challenge on Large-Scale Biomedical Semantic Indexing and Question Answering](https://mlsanity.com/paper/2210.06852/) (2022)
- [Classifying Scientific Publications with BERT -- Is Self-Attention a Feature Selection Method?](https://mlsanity.com/paper/2101.08114/) (2021)
- [Beyond MeSH: Fine-Grained Semantic Indexing of Biomedical Literature based on Weak Supervision](https://mlsanity.com/paper/2005.07638/) (2020)
- [What is all this new MeSH about? Exploring the semantic provenance of new descriptors in the MeSH thesaurus](https://mlsanity.com/paper/2101.08293/) (2021)

## Abstract (arXiv)

Objective: Semantic indexing of biomedical literature is usually done at the level of MeSH descriptors with several related but distinct biomedical concepts often grouped together and treated as a single topic. This study proposes a new method for the automated refinement of subject annotations at the level of MeSH concepts. Methods: Lacking labelled data, we rely on weak supervision based on concept occurrence in the abstract of an article, which is also enhanced by dictionary-based heuristics. In addition, we investigate deep learning approaches, making design choices to tackle the particular challenges of this task. The new method is evaluated on a large-scale retrospective scenario, based on concepts that have been promoted to descriptors. Results: In our experiments concept occurrence was the strongest heuristic achieving a macro-F1 score of about 0.63 across several labels. The proposed method improved it further by more than 4pp. Conclusion: The results suggest that concept occurrence is a strong heuristic for refining the coarse-grained labels at the level of MeSH concepts and the proposed method improves it further.