# Learning Cross-Context Entity Representations from Text

arXiv: https://arxiv.org/abs/2001.03765 · HTML: https://mlsanity.com/paper/2001.03765/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jan 2020.

## Built on by (most cited first, 12 of 12)

- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [Relational World Knowledge Representation in Contextual Language Models: A Review](https://mlsanity.com/paper/2104.05837/) (2021)
- [EASE: Entity-Aware Contrastive Learning of Sentence Embedding](https://mlsanity.com/paper/2205.04260/) (2022)
- [Reasoning Over Virtual Knowledge Bases With Open Predicate Relations](https://mlsanity.com/paper/2102.07043/) (2021)
- [DOCENT: Learning Self-Supervised Entity Representations from Large Document Collections](https://mlsanity.com/paper/2102.13247/) (2021)
- [Improving Candidate Retrieval with Entity Profile Generation for Wikidata Entity Linking](https://mlsanity.com/paper/2202.13404/) (2022)
- [MuVER: Improving First-Stage Entity Retrieval with Multi-View Entity Representations](https://mlsanity.com/paper/2109.05716/) (2021)
- [A Unified Encoder-Decoder Framework with Entity Memory](https://mlsanity.com/paper/2210.03273/) (2022)
- [What's the best place for an AI conference, Vancouver or ______: Why completing comparative questions is difficult](https://mlsanity.com/paper/2104.01940/) (2021)
- [Hansel: A Chinese Few-Shot and Zero-Shot Entity Linking Benchmark](https://mlsanity.com/paper/2207.13005/) (2022)
- [Entity-Centric Query Refinement](https://mlsanity.com/paper/2204.00743/) (2022)
- [MeKB-Rec: Personal Knowledge Graph Learning for Cross-Domain Recommendation](https://mlsanity.com/paper/2310.11088/) (2023)

## Abstract (arXiv)

Language modeling tasks, in which words, or word-pieces, are predicted on the basis of a local context, have been very effective for learning word embeddings and context dependent representations of phrases. Motivated by the observation that efforts to code world knowledge into machine readable knowledge bases or human readable encyclopedias tend to be entity-centric, we investigate the use of a fill-in-the-blank task to learn context independent representations of entities from the text contexts in which those entities were mentioned. We show that large scale training of neural models allows us to learn high quality entity representations, and we demonstrate successful results on four domains: (1) existing entity-level typing benchmarks, including a 64% error reduction over previous work on TypeNet (Murty et al., 2018); (2) a novel few-shot category reconstruction task; (3) existing entity linking benchmarks, where we match the state-of-the-art on CoNLL-Aida without linking-specific features and obtain a score of 89.8% on TAC-KBP 2010 without using any alias table, external knowledge base or in domain training data and (4) answering trivia questions, which uniquely identify entities. Our global entity representations encode fine-grained type categories, such as Scottish footballers, and can answer trivia questions such as: Who was the last inmate of Spandau jail in Berlin?