# Modeling Fine-Grained Entity Types with Box Embeddings

arXiv: https://arxiv.org/abs/2101.00345 · HTML: https://mlsanity.com/paper/2101.00345/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 57 references that are in the map.
- First posted on arXiv 2 Jan 2021.

## Builds on (most cited first, 30 of 35)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Poincar'e Embeddings for Learning Hierarchical Representations](https://mlsanity.com/paper/1705.08039/) (2017)
- [Calibration of Pre-trained Transformers](https://mlsanity.com/paper/2003.07892/) (2020)
- [Highway Networks](https://mlsanity.com/paper/1505.00387/) (2015)
- [Query2box: Reasoning over Knowledge Graphs in Vector Space using Box Embeddings](https://mlsanity.com/paper/2002.05969/) (2020)
- [Hyperbolic Entailment Cones for Learning Hierarchical Embeddings](https://mlsanity.com/paper/1804.01882/) (2018)
- [Order-Embeddings of Images and Language](https://mlsanity.com/paper/1511.06361/) (2015)
- [BoxE: A Box Embedding Model for Knowledge Base Completion](https://mlsanity.com/paper/2007.06267/) (2020)
- [Word Representations via Gaussian Embedding](https://mlsanity.com/paper/1412.6623/) (2014)
- [Ultra-Fine Entity Typing](https://mlsanity.com/paper/1807.04905/) (2018)
- [Posterior calibration and exploratory analysis for natural language processing models](https://mlsanity.com/paper/1508.05154/) (2015)
- [Probabilistic Embedding of Knowledge Graphs with Box Lattice Measures](https://mlsanity.com/paper/1805.06627/) (2018)
- [Fine-Grained Entity Typing for Domain Independent Entity Linking](https://mlsanity.com/paper/1909.05780/) (2019)
- [Improving Local Identifiability in Probabilistic Box Embeddings](https://mlsanity.com/paper/2010.04831/) (2020)
- [Context-Dependent Fine-Grained Entity Type Tagging](https://mlsanity.com/paper/1412.1820/) (2014)
- [PreCo: A Large-scale Dataset in Preschool Vocabulary for Coreference Resolution](https://mlsanity.com/paper/1810.09807/) (2018)
- [Learning to Denoise Distantly-Labeled Data for Entity Typing](https://mlsanity.com/paper/1905.01566/) (2019)
- [A Fully Hyperbolic Neural Model for Hierarchical Multi-Class Classification](https://mlsanity.com/paper/2010.02053/) (2020)
- [Label Noise Reduction in Entity Typing by Heterogeneous Partial-Label Embedding](https://mlsanity.com/paper/1602.05307/) (2016)
- [Imposing Label-Relational Inductive Bias for Extremely Fine-Grained Entity Typing](https://mlsanity.com/paper/1903.02591/) (2019)
- [Neural Fine-Grained Entity Type Classification with Hierarchy-Aware Loss](https://mlsanity.com/paper/1803.03378/) (2018)
- [Interpretable Entity Representations through Large-Scale Typing](https://mlsanity.com/paper/2005.00147/) (2020)
- [Hierarchical Losses and New Resources for Fine-grained Entity Typing and Linking](https://mlsanity.com/paper/1807.05127/) (2018)
- [Hierarchical Entity Typing via Multi-level Learning to Rank](https://mlsanity.com/paper/2004.02286/) (2020)

## Built on by (most cited first, 30 of 36)

- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023)
- [Prompt-Learning for Fine-Grained Entity Typing](https://mlsanity.com/paper/2108.10604/) (2021)
- [Ultra-Fine Entity Typing with Weak Supervision from a Masked Language Model](https://mlsanity.com/paper/2106.04098/) (2021)
- [Unified Semantic Typing with Meaningful Label Inference](https://mlsanity.com/paper/2205.01826/) (2022)
- [Ultra-fine Entity Typing with Indirect Supervision from Natural Language Inference](https://mlsanity.com/paper/2202.06167/) (2022)
- [Acquiring and Modelling Abstract Commonsense Knowledge via Conceptualization](https://mlsanity.com/paper/2206.01532/) (2022)
- [Does Your Model Classify Entities Reasonably? Diagnosing and Mitigating Spurious Correlations in Entity Typing](https://mlsanity.com/paper/2205.12640/) (2022)
- [Automatic Noisy Label Correction for Fine-Grained Entity Typing](https://mlsanity.com/paper/2205.03011/) (2022)
- [Transformer-based Entity Typing in Knowledge Graphs](https://mlsanity.com/paper/2210.11151/) (2022)
- [Word2Box: Capturing Set-Theoretic Semantics of Words using Box Embeddings](https://mlsanity.com/paper/2106.14361/) (2021)
- [Box Embeddings: An open-source library for representation learning using geometric structures](https://mlsanity.com/paper/2109.04997/) (2021)
- [Geometric Relational Embeddings: A Survey](https://mlsanity.com/paper/2304.11949/) (2023)
- [TABi: Type-Aware Bi-Encoders for Open-Domain Entity Retrieval](https://mlsanity.com/paper/2204.08173/) (2022)
- [Generative Entity Typing with Curriculum Learning](https://mlsanity.com/paper/2210.02914/) (2022)
- [Logical Entity Representation in Knowledge-Graphs for Differentiable Rule Learning](https://mlsanity.com/paper/2305.12738/) (2023)
- [Ultra-Fine Entity Typing with Prior Knowledge about Labels: A Simple Clustering Based Strategy](https://mlsanity.com/paper/2305.12802/) (2023)
- [Concept2Box: Joint Geometric Embeddings for Learning Two-View Knowledge Graphs](https://mlsanity.com/paper/2307.01933/) (2023)
- [Representing Spatial Trajectories as Distributions](https://mlsanity.com/paper/2210.01322/) (2022)
- [Calibrated Seq2seq Models for Efficient and Generalizable Ultra-fine Entity Typing](https://mlsanity.com/paper/2311.00835/) (2023)
- [Learning to Select from Multiple Options](https://mlsanity.com/paper/2212.00301/) (2022)
- [Modeling Label Correlations for Ultra-Fine Entity Typing with Neural Pairwise Conditional Random Field](https://mlsanity.com/paper/2212.01581/) (2022)
- [Recall, Expand and Multi-Candidate Cross-Encode: Fast and Accurate Ultra-Fine Entity Typing](https://mlsanity.com/paper/2212.09125/) (2022)
- [Dense Retrieval as Indirect Supervision for Large-space Decision Making](https://mlsanity.com/paper/2310.18619/) (2023)
- [On the Role of Entity and Event Level Conceptualization in Generalizable Reasoning: A Survey of Tasks, Methods, Applications, and Future Directions](https://mlsanity.com/paper/2406.10885/) (2024)
- [All Entities are Not Created Equal: Examining the Long Tail for Ultra-Fine Entity Typing](https://mlsanity.com/paper/2410.17355/) (2024)
- [Type-enriched Hierarchical Contrastive Strategy for Fine-Grained Entity Typing](https://mlsanity.com/paper/2208.10081/) (2022)
- [EnCore: Fine-Grained Entity Typing by Pre-Training Entity Encoders on Coreference Chains](https://mlsanity.com/paper/2305.12924/) (2023)
- [From Ultra-Fine to Fine: Fine-tuning Ultra-Fine Entity Typing Models to Fine-grained](https://mlsanity.com/paper/2312.06188/) (2023)
- [Set2Box: Similarity Preserving Representation Learning of Sets](https://mlsanity.com/paper/2210.03282/) (2022)
- [Denoising Enhanced Distantly Supervised Ultrafine Entity Typing](https://mlsanity.com/paper/2210.09599/) (2022)

## Abstract (arXiv)

Neural entity typing models typically represent fine-grained entity types as vectors in a high-dimensional space, but such spaces are not well-suited to modeling these types' complex interdependencies. We study the ability of box embeddings, which embed concepts as d-dimensional hyperrectangles, to capture hierarchies of types even when these relationships are not defined explicitly in the ontology. Our model represents both types and entity mentions as boxes. Each mention and its context are fed into a BERT-based model to embed that mention in our box space; essentially, this model leverages typological clues present in the surface text to hypothesize a type representation for the mention. Box containment can then be used to derive both the posterior probability of a mention exhibiting a given type and the conditional probability relations between types themselves. We compare our approach with a vector-based typing model and observe state-of-the-art performance on several entity typing benchmarks. In addition to competitive typing performance, our box-based model shows better performance in prediction consistency (predicting a supertype and a subtype together) and confidence (i.e., calibration), demonstrating that the box-based model captures the latent type hierarchies better than the vector-based model does.