# Zero-shot Entity Linking with Efficient Long Range Sequence Modeling

arXiv: https://arxiv.org/abs/2010.06065 · HTML: https://mlsanity.com/paper/2010.06065/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Oct 2020.

## Built on by (most cited first, 13 of 13)

- [Extracting Biomedical Factual Knowledge Using Pretrained Language Model and Electronic Health Record Context](https://mlsanity.com/paper/2209.07859/) (2022)
- [Context Variance Evaluation of Pretrained Language Models for Prompt-based Biomedical Knowledge Probing](https://mlsanity.com/paper/2211.10265/) (2022)
- [UMASS_BioNLP at MEDIQA-Chat 2023: Can LLMs generate high-quality synthetic note-oriented doctor-patient conversations?](https://mlsanity.com/paper/2306.16931/) (2023)
- [Improving Formality Style Transfer with Context-Aware Rule Injection](https://mlsanity.com/paper/2106.00210/) (2021)
- [MuVER: Improving First-Stage Entity Retrieval with Multi-View Entity Representations](https://mlsanity.com/paper/2109.05716/) (2021)
- [Modeling Fine-grained Information via Knowledge-aware Hierarchical Graph for Zero-shot Entity Retrieval](https://mlsanity.com/paper/2211.10991/) (2022)
- [Improving Zero-Shot Entity Retrieval through Effective Dense Representations](https://mlsanity.com/paper/2103.04156/) (2021)
- [Automated Identification of Eviction Status from Electronic Health Record Notes](https://mlsanity.com/paper/2212.02762/) (2022)
- [A Read-and-Select Framework for Zero-shot Entity Linking](https://mlsanity.com/paper/2310.12450/) (2023)
- [LELA: an LLM-based Entity Linking Approach with Zero-Shot Domain Adaptation](https://mlsanity.com/paper/2601.05192/) (2026)
- [Towards Better Entity Linking with Multi-View Enhanced Distillation](https://mlsanity.com/paper/2305.17371/) (2023)
- [Hybrid X-Linker: Automated Data Generation and Extreme Multi-label Ranking for Biomedical Entity Linking](https://mlsanity.com/paper/2407.06292/) (2024)
- [CYCLE: Cross-Year Contrastive Learning in Entity-Linking](https://mlsanity.com/paper/2410.09127/) (2024)

## Abstract (arXiv)

This paper considers the problem of zero-shot entity linking, in which a link in the test time may not present in training. Following the prevailing BERT-based research efforts, we find a simple yet effective way is to expand the long-range sequence modeling. Unlike many previous methods, our method does not require expensive pre-training of BERT with long position embedding. Instead, we propose an efficient position embeddings initialization method called Embedding-repeat, which initializes larger position embeddings based on BERT-Base. On Wikia's zero-shot EL dataset, our method improves the SOTA from 76.06% to 79.08%, and for its long data, the corresponding improvement is from 74.57% to 82.14%. Our experiments suggest the effectiveness of long-range sequence modeling without retraining the BERT model.