# Recall, Expand and Multi-Candidate Cross-Encode: Fast and Accurate Ultra-Fine Entity Typing

arXiv: https://arxiv.org/abs/2212.09125 · HTML: https://mlsanity.com/paper/2212.09125/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 31 references that are in the map.
- First posted on arXiv 18 Dec 2022.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation](https://mlsanity.com/paper/1810.10147/) (2018)
- [Ultra-Fine Entity Typing](https://mlsanity.com/paper/1807.04905/) (2018)
- [Scalable Zero-shot Entity Linking with Dense Entity Retrieval](https://mlsanity.com/paper/1911.03814/) (2019)
- [Prompt-Learning for Fine-Grained Entity Typing](https://mlsanity.com/paper/2108.10604/) (2021)
- [NEZHA: Neural Contextualized Representation for Chinese Language Understanding](https://mlsanity.com/paper/1909.00204/) (2019)
- [Ultra-Fine Entity Typing with Weak Supervision from a Masked Language Model](https://mlsanity.com/paper/2106.04098/) (2021)
- [Modeling Fine-Grained Entity Types with Box Embeddings](https://mlsanity.com/paper/2101.00345/) (2021)
- [Learning to Denoise Distantly-Labeled Data for Entity Typing](https://mlsanity.com/paper/1905.01566/) (2019)
- [Building Legal Case Retrieval Systems with Lexical Matching and Summarization using A Pre-Trained Phrase Scoring Model](https://mlsanity.com/paper/2009.14083/) (2020)
- [Imposing Label-Relational Inductive Bias for Extremely Fine-Grained Entity Typing](https://mlsanity.com/paper/1903.02591/) (2019)
- [Fine-grained Entity Typing via Label Reasoning](https://mlsanity.com/paper/2109.05744/) (2021)
- [HLATR: Enhance Multi-stage Text Retrieval with Hybrid List Aware Transformer Reranking](https://mlsanity.com/paper/2205.10569/) (2022)
- [Ultra-fine Entity Typing with Indirect Supervision from Natural Language Inference](https://mlsanity.com/paper/2202.06167/) (2022)
- [Automatic Noisy Label Correction for Fine-Grained Entity Typing](https://mlsanity.com/paper/2205.03011/) (2022)
- [A Chinese Corpus for Fine-grained Entity Typing](https://mlsanity.com/paper/2004.08825/) (2020)
- [Learning to Select from Multiple Options](https://mlsanity.com/paper/2212.00301/) (2022)
- [Modeling Label Correlations for Ultra-Fine Entity Typing with Neural Pairwise Conditional Random Field](https://mlsanity.com/paper/2212.01581/) (2022)

## Built on by (most cited first, 4 of 4)

- [Unsupervised Entity Alignment for Temporal Knowledge Graphs](https://mlsanity.com/paper/2302.00796/) (2023)
- [On the Role of Entity and Event Level Conceptualization in Generalizable Reasoning: A Survey of Tasks, Methods, Applications, and Future Directions](https://mlsanity.com/paper/2406.10885/) (2024)
- [OntoType: Ontology-Guided and Pre-Trained Language Model Assisted Fine-Grained Entity Typing](https://mlsanity.com/paper/2305.12307/) (2023)
- [From Ultra-Fine to Fine: Fine-tuning Ultra-Fine Entity Typing Models to Fine-grained](https://mlsanity.com/paper/2312.06188/) (2023)

## Abstract (arXiv)

Ultra-fine entity typing (UFET) predicts extremely free-formed types (e.g., president, politician) of a given entity mention (e.g., Joe Biden) in context. State-of-the-art (SOTA) methods use the cross-encoder (CE) based architecture. CE concatenates the mention (and its context) with each type and feeds the pairs into a pretrained language model (PLM) to score their relevance. It brings deeper interaction between mention and types to reach better performance but has to perform N (type set size) forward passes to infer types of a single mention. CE is therefore very slow in inference when the type set is large (e.g., N = 10k for UFET). To this end, we propose to perform entity typing in a recall-expand-filter manner. The recall and expand stages prune the large type set and generate K (K is typically less than 256) most relevant type candidates for each mention. At the filter stage, we use a novel model called MCCE to concurrently encode and score these K candidates in only one forward pass to obtain the final type prediction. We investigate different variants of MCCE and extensive experiments show that MCCE under our paradigm reaches SOTA performance on ultra-fine entity typing and is thousands of times faster than the cross-encoder. We also found MCCE is very effective in fine-grained (130 types) and coarse-grained (9 types) entity typing. Our code is available at urlhttps://github.com/modelscope/AdaSeq/tree/master/examples/MCCE.