# Investigating Entity Knowledge in BERT with Simple Neural End-To-End Entity Linking

arXiv: https://arxiv.org/abs/2003.05473 · HTML: https://mlsanity.com/paper/2003.05473/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Mar 2020.

## Built on by (most cited first, 30 of 49)

- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [CM3: A Causal Masked Multimodal Model of the Internet](https://mlsanity.com/paper/2201.07520/) (2022)
- [ReFinED: An Efficient Zero-shot-capable Approach to End-to-End Entity Linking](https://mlsanity.com/paper/2207.04108/) (2022)
- [EntQA: Entity Linking as Question Answering](https://mlsanity.com/paper/2110.02369/) (2021)
- [Relational World Knowledge Representation in Contextual Language Models: A Review](https://mlsanity.com/paper/2104.05837/) (2021)
- [Entity Linking Meets Deep Learning: Techniques and Solutions](https://mlsanity.com/paper/2109.12520/) (2021)
- [Highly Parallel Autoregressive Entity Linking with Discriminative Correction](https://mlsanity.com/paper/2109.03792/) (2021)
- [COPEN: Probing Conceptual Knowledge in Pre-trained Language Models](https://mlsanity.com/paper/2211.04079/) (2022)
- [XLM-K: Improving Cross-Lingual Language Model Pre-training with Multilingual Knowledge](https://mlsanity.com/paper/2109.12573/) (2021)
- [Instructed Language Models with Retrievers Are Powerful Entity Linkers](https://mlsanity.com/paper/2311.03250/) (2023)
- [ReLiK: Retrieve and LinK, Fast and Accurate Entity Linking and Relation Extraction on an Academic Budget](https://mlsanity.com/paper/2408.00103/) (2024)
- [DKPLM: Decomposable Knowledge-enhanced Pre-trained Language Model for Natural Language Understanding](https://mlsanity.com/paper/2112.01047/) (2021)
- [CHOLAN: A Modular Approach for Neural Entity Linking on Wikipedia and Wikidata](https://mlsanity.com/paper/2101.09969/) (2021)
- [OneGen: Efficient One-Pass Unified Generation and Retrieval for LLMs](https://mlsanity.com/paper/2409.05152/) (2024)
- [Detection, Disambiguation, Re-ranking: Autoregressive Entity Linking as a Multi-Task Problem](https://mlsanity.com/paper/2204.05990/) (2022)
- [SpEL: Structured Prediction for Entity Linking](https://mlsanity.com/paper/2310.14684/) (2023)
- [Entity-aware Transformers for Entity Search](https://mlsanity.com/paper/2205.00820/) (2022)
- [Kuaipedia: a Large-scale Multi-modal Short-video Encyclopedia](https://mlsanity.com/paper/2211.00732/) (2022)
- [Multilingual End to End Entity Linking](https://mlsanity.com/paper/2306.08896/) (2023)
- [Reddit Entity Linking Dataset](https://mlsanity.com/paper/2101.01228/) (2021)
- [Bidirectional End-to-End Learning of Retriever-Reader Paradigm for Entity Linking](https://mlsanity.com/paper/2306.12245/) (2023)
- [Towards Consistent Document-level Entity Linking: Joint Models for Entity Linking and Coreference Resolution](https://mlsanity.com/paper/2108.13530/) (2021)
- [Knowledge-augmented Few-shot Visual Relation Detection](https://mlsanity.com/paper/2303.05342/) (2023)
- [Benchmarking Diverse-Modal Entity Linking with Generative Models](https://mlsanity.com/paper/2305.17337/) (2023)
- [How Proficient Are Large Language Models in Formal Languages? An In-Depth Insight for Knowledge Base Question Answering](https://mlsanity.com/paper/2401.05777/) (2024)
- [Evaluating Design Decisions for Dual Encoder-based Entity Disambiguation](https://mlsanity.com/paper/2505.11683/) (2025)
- [Neural Generation Meets Real People: Building a Social, Informative Open-Domain Dialogue Agent](https://mlsanity.com/paper/2207.12021/) (2022)
- [Entity Disambiguation via Fusion Entity Decoding](https://mlsanity.com/paper/2404.01626/) (2024)
- [AR-BERT: Aspect-relation enhanced Aspect-level Sentiment Classification with Multi-modal Explanations](https://mlsanity.com/paper/2108.11656/) (2021)

## Abstract (arXiv)

A typical architecture for end-to-end entity linking systems consists of three steps: mention detection, candidate generation and entity disambiguation. In this study we investigate the following questions: (a) Can all those steps be learned jointly with a model for contextualized text-representations, i.e. BERT (Devlin et al., 2019)? (b) How much entity knowledge is already contained in pretrained BERT? (c) Does additional entity knowledge improve BERT's performance in downstream tasks? To this end, we propose an extreme simplification of the entity linking setup that works surprisingly well: simply cast it as a per token classification over the entire entity vocabulary (over 700K classes in our case). We show on an entity linking benchmark that (i) this model improves the entity representations over plain BERT, (ii) that it outperforms entity linking architectures that optimize the tasks separately and (iii) that it only comes second to the current state-of-the-art that does mention detection and entity disambiguation jointly. Additionally, we investigate the usefulness of entity-aware token-representations in the text-understanding benchmark GLUE, as well as the question answering benchmarks SQUAD V2 and SWAG and also the EN-DE WMT14 machine translation benchmark. To our surprise, we find that most of those benchmarks do not benefit from additional entity knowledge, except for a task with very small training data, the RTE task in GLUE, which improves by 2%.