# A Fair and In-Depth Evaluation of Existing End-to-End Entity Linking Systems

arXiv: https://arxiv.org/abs/2305.14937 · HTML: https://mlsanity.com/paper/2305.14937/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 40 references that are in the map.
- First posted on arXiv 24 May 2023.

## Builds on (most cited first, 9 of 9)

- [Autoregressive Entity Retrieval](https://mlsanity.com/paper/2010.00904/) (2020)
- [End-to-End Neural Entity Linking](https://mlsanity.com/paper/1808.07699/) (2018)
- [ReFinED: An Efficient Zero-shot-capable Approach to End-to-End Entity Linking](https://mlsanity.com/paper/2207.04108/) (2022)
- [REL: An Entity Linker Standing on the Shoulders of Giants](https://mlsanity.com/paper/2006.01969/) (2020)
- [Investigating Entity Knowledge in BERT with Simple Neural End-To-End Entity Linking](https://mlsanity.com/paper/2003.05473/) (2020)
- [Highly Parallel Autoregressive Entity Linking with Discriminative Correction](https://mlsanity.com/paper/2109.03792/) (2021)
- [Analysis of Named Entity Recognition and Linking for Tweets](https://mlsanity.com/paper/1410.7182/) (2014)
- [CHOLAN: A Modular Approach for Neural Entity Linking on Wikipedia and Wikidata](https://mlsanity.com/paper/2101.09969/) (2021)
- [ELEVANT: A Fully Automatic Fine-Grained Entity Linking Evaluation and Analysis Tool](https://mlsanity.com/paper/2208.07193/) (2022)

## Built on by (most cited first, 2 of 2)

- [Enriching Social Science Research via Survey Item Linking](https://mlsanity.com/paper/2412.15831/) (2024)
- [CADEL: A Corpus of Administrative Web Documents for Japanese Entity Linking](https://mlsanity.com/paper/2603.29336/) (2026)

## Abstract (arXiv)

Existing evaluations of entity linking systems often say little about how the system is going to perform for a particular application. There are two fundamental reasons for this. One is that many evaluations only use aggregate measures (like precision, recall, and F1 score), without a detailed error analysis or a closer look at the results. The other is that all of the widely used benchmarks have strong biases and artifacts, in particular: a strong focus on named entities, an unclear or missing specification of what else counts as an entity mention, poor handling of ambiguities, and an over- or underrepresentation of certain kinds of entities. We provide a more meaningful and fair in-depth evaluation of a variety of existing end-to-end entity linkers. We characterize their strengths and weaknesses and also report on reproducibility aspects. The detailed results of our evaluation can be inspected under https://elevant.cs.uni-freiburg.de/emnlp2023 . Our evaluation is based on several widely used benchmarks, which exhibit the problems mentioned above to various degrees, as well as on two new benchmarks, which address the problems mentioned above. The new benchmarks can be found under https://github.com/ad-freiburg/fair-entity-linking-benchmarks .