# All Entities are Not Created Equal: Examining the Long Tail for Ultra-Fine Entity Typing

arXiv: https://arxiv.org/abs/2410.17355 · HTML: https://mlsanity.com/paper/2410.17355/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 28 references that are in the map.
- First posted on arXiv 22 Oct 2024.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens](https://mlsanity.com/paper/2401.17377/) (2024)
- [Ultra-Fine Entity Typing](https://mlsanity.com/paper/1807.04905/) (2018)
- [Ultra-Fine Entity Typing with Weak Supervision from a Masked Language Model](https://mlsanity.com/paper/2106.04098/) (2021)
- [Fine-Grained Entity Typing for Domain Independent Entity Linking](https://mlsanity.com/paper/1909.05780/) (2019)
- [Modeling Fine-Grained Entity Types with Box Embeddings](https://mlsanity.com/paper/2101.00345/) (2021)
- [Context-Dependent Fine-Grained Entity Type Tagging](https://mlsanity.com/paper/1412.1820/) (2014)
- [Fine-grained Entity Typing via Label Reasoning](https://mlsanity.com/paper/2109.05744/) (2021)
- [Noise Mitigation for Neural Entity Typing and Relation Extraction](https://mlsanity.com/paper/1612.07495/) (2016)
- [Automatic Noisy Label Correction for Fine-Grained Entity Typing](https://mlsanity.com/paper/2205.03011/) (2022)
- [Ultra-Fine Entity Typing with Prior Knowledge about Labels: A Simple Clustering Based Strategy](https://mlsanity.com/paper/2305.12802/) (2023)
- [Too Brittle To Touch: Comparing the Stability of Quantization and Distillation Towards Developing Lightweight Low-Resource MT Models](https://mlsanity.com/paper/2210.15184/) (2022)

## Built on by (most cited first, 3 of 3)

- [MMKG-RDS: Reasoning Data Synthesis via Deep Mining of Multimodal Knowledge Graphs](https://mlsanity.com/paper/2602.23632/) (2026)
- [Narrative-UFET: Narrative Generation for Ultra-Fine Entity Typing](https://mlsanity.com/paper/2606.27598/) (2026)
- [Toward a Gricean Retreat: Probing LLMs for Knowledge Boundaries and Referent Specificity](https://mlsanity.com/paper/2608.13484/) (2026)

## Abstract (arXiv)

Due to their capacity to acquire world knowledge from large corpora, pre-trained language models (PLMs) are extensively used in ultra-fine entity typing tasks where the space of labels is extremely large. In this work, we explore the limitations of the knowledge acquired by PLMs by proposing a novel heuristic to approximate the pre-training distribution of entities when the pre-training data is unknown. Then, we systematically demonstrate that entity-typing approaches that rely solely on the parametric knowledge of PLMs struggle significantly with entities at the long tail of the pre-training distribution, and that knowledge-infused approaches can account for some of these shortcomings. Our findings suggest that we need to go beyond PLMs to produce solutions that perform well for infrequent entities.