# BERT might be Overkill: A Tiny but Effective Biomedical Entity Linker based on Residual Convolutional Neural Networks

arXiv: https://arxiv.org/abs/2109.02237 · HTML: https://mlsanity.com/paper/2109.02237/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 45 references that are in the map.
- First posted on arXiv 6 Sep 2021.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Self-Alignment Pretraining for Biomedical Entity Representations](https://mlsanity.com/paper/2010.11784/) (2020)
- [Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little](https://mlsanity.com/paper/2104.06644/) (2021)
- [Learning Dense Representations for Entity Retrieval](https://mlsanity.com/paper/1909.10506/) (2019)
- [Scalable Zero-shot Entity Linking with Dense Entity Retrieval](https://mlsanity.com/paper/1911.03814/) (2019)
- [MedMentions: A Large Biomedical Corpus Annotated with UMLS Concepts](https://mlsanity.com/paper/1902.09476/) (2019)
- [Out of Order: How Important Is The Sequential Order of Words in a Sentence in Natural Language Understanding Tasks?](https://mlsanity.com/paper/2012.15180/) (2020)
- [Biomedical Entity Representations with Synonym Marginalization](https://mlsanity.com/paper/2005.00239/) (2020)
- [COMETA: A Corpus for Medical Entity Linking in the Social Media](https://mlsanity.com/paper/2010.03295/) (2020)
- [COVID-19 Literature Knowledge Graph Construction and Drug Repurposing Report Generation](https://mlsanity.com/paper/2007.00576/) (2020)
- [BERT-based Ranking for Biomedical Entity Normalization](https://mlsanity.com/paper/1908.03548/) (2019)
- [Learning Domain-Specialised Representations for Cross-Lingual Biomedical Entity Linking](https://mlsanity.com/paper/2105.14398/) (2021)
- [Joint Biomedical Entity and Relation Extraction with Knowledge-Enhanced Collective Inference](https://mlsanity.com/paper/2105.13456/) (2021)
- [A Lightweight Neural Model for Biomedical Entity Linking](https://mlsanity.com/paper/2012.08844/) (2020)
- [Answering Questions on COVID-19 in Real-Time](https://mlsanity.com/paper/2006.15830/) (2020)
- [A Context-Dependent Gated Module for Incorporating Symbolic Semantics into Event Coreference Resolution](https://mlsanity.com/paper/2104.01697/) (2021)
- [Deep Residual Learning for Weakly-Supervised Relation Extraction](https://mlsanity.com/paper/1707.08866/) (2017)
- [A Joint Learning Approach based on Self-Distillation for Keyphrase Extraction from Scientific Documents](https://mlsanity.com/paper/2010.11980/) (2020)

## Built on by (most cited first, 14 of 14)

- [BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model](https://mlsanity.com/paper/2204.03905/) (2022)
- [Knowledge Injected Prompt Based Fine-tuning for Multi-label Few-shot ICD Coding](https://mlsanity.com/paper/2210.03304/) (2022)
- [Generative Biomedical Entity Linking via Knowledge Base-Guided Pre-training and Synonyms-Aware Fine-tuning](https://mlsanity.com/paper/2204.05164/) (2022)
- [Knowledge-Rich Self-Supervision for Biomedical Entity Linking](https://mlsanity.com/paper/2112.07887/) (2021)
- [Detection, Disambiguation, Re-ranking: Autoregressive Entity Linking as a Multi-Task Problem](https://mlsanity.com/paper/2204.05990/) (2022)
- [Improving Candidate Retrieval with Entity Profile Generation for Wikidata Entity Linking](https://mlsanity.com/paper/2202.13404/) (2022)
- [BELB: a Biomedical Entity Linking Benchmark](https://mlsanity.com/paper/2308.11537/) (2023)
- [Learning from Negative Samples in Biomedical Generative Entity Linking](https://mlsanity.com/paper/2408.16493/) (2024)
- [BELHD: Improving Biomedical Entity Linking with Homonoym Disambiguation](https://mlsanity.com/paper/2401.05125/) (2024)
- [Exploring Partial Knowledge Base Inference in Biomedical Entity Linking](https://mlsanity.com/paper/2303.10330/) (2023)
- [Biomedical Entity Linking with Triple-aware Pre-Training](https://mlsanity.com/paper/2308.14429/) (2023)
- [Improving Biomedical Entity Linking with Retrieval-enhanced Learning](https://mlsanity.com/paper/2312.09806/) (2023)
- [Biomedical Entity Linking as Multiple Choice Question Answering](https://mlsanity.com/paper/2402.15189/) (2024)
- [Neighborhood-Aware Dual Biomedical Entity Linking](https://mlsanity.com/paper/2608.04144/) (2026)

## Abstract (arXiv)

Biomedical entity linking is the task of linking entity mentions in a biomedical document to referent entities in a knowledge base. Recently, many BERT-based models have been introduced for the task. While these models have achieved competitive results on many datasets, they are computationally expensive and contain about 110M parameters. Little is known about the factors contributing to their impressive performance and whether the over-parameterization is needed. In this work, we shed some light on the inner working mechanisms of these large BERT-based models. Through a set of probing experiments, we have found that the entity linking performance only changes slightly when the input word order is shuffled or when the attention scope is limited to a fixed window size. From these observations, we propose an efficient convolutional neural network with residual connections for biomedical entity linking. Because of the sparse connectivity and weight sharing properties, our model has a small number of parameters and is highly efficient. On five public datasets, our model achieves comparable or even better linking accuracy than the state-of-the-art BERT-based models while having about 60 times fewer parameters.