# Modeling Label Correlations for Ultra-Fine Entity Typing with Neural Pairwise Conditional Random Field

arXiv: https://arxiv.org/abs/2212.01581 · HTML: https://mlsanity.com/paper/2212.01581/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 52 references that are in the map.
- First posted on arXiv 3 Dec 2022.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [Conditional Random Fields as Recurrent Neural Networks](https://mlsanity.com/paper/1502.03240/) (2015)
- [Few-NERD: A Few-Shot Named Entity Recognition Dataset](https://mlsanity.com/paper/2105.07464/) (2021)
- [Ultra-Fine Entity Typing](https://mlsanity.com/paper/1807.04905/) (2018)
- [Prompt-Learning for Fine-Grained Entity Typing](https://mlsanity.com/paper/2108.10604/) (2021)
- [Ultra-Fine Entity Typing with Weak Supervision from a Masked Language Model](https://mlsanity.com/paper/2106.04098/) (2021)
- [Modeling Fine-Grained Entity Types with Box Embeddings](https://mlsanity.com/paper/2101.00345/) (2021)
- [Context-Dependent Fine-Grained Entity Type Tagging](https://mlsanity.com/paper/1412.1820/) (2014)
- [Fast, Exact and Multi-Scale Inference for Semantic Image Segmentation with Deep Gaussian CRFs](https://mlsanity.com/paper/1603.08358/) (2016)
- [Higher Order Conditional Random Fields in Deep Neural Networks](https://mlsanity.com/paper/1511.08119/) (2015)
- [Learning to Denoise Distantly-Labeled Data for Entity Typing](https://mlsanity.com/paper/1905.01566/) (2019)
- [A Fully Hyperbolic Neural Model for Hierarchical Multi-Class Classification](https://mlsanity.com/paper/2010.02053/) (2020)
- [Label Noise Reduction in Entity Typing by Heterogeneous Partial-Label Embedding](https://mlsanity.com/paper/1602.05307/) (2016)
- [Imposing Label-Relational Inductive Bias for Extremely Fine-Grained Entity Typing](https://mlsanity.com/paper/1903.02591/) (2019)
- [Fine-grained Entity Typing via Label Reasoning](https://mlsanity.com/paper/2109.05744/) (2021)
- [Neural Fine-Grained Entity Type Classification with Hierarchy-Aware Loss](https://mlsanity.com/paper/1803.03378/) (2018)
- [Label Semantic Aware Pre-training for Few-shot Text Classification](https://mlsanity.com/paper/2204.07128/) (2022)
- [Ultra-fine Entity Typing with Indirect Supervision from Natural Language Inference](https://mlsanity.com/paper/2202.06167/) (2022)
- [Automatic Noisy Label Correction for Fine-Grained Entity Typing](https://mlsanity.com/paper/2205.03011/) (2022)
- [Regularized Frank-Wolfe for Dense CRFs: Generalizing Mean Field and Beyond](https://mlsanity.com/paper/2110.14759/) (2021)
- [Regularizing Model Complexity and Label Structure for Multi-Label Text Classification](https://mlsanity.com/paper/1705.00740/) (2017)
- [An Investigation of Potential Function Designs for Neural CRF](https://mlsanity.com/paper/2011.05604/) (2020)

## Built on by (most cited first, 4 of 4)

- [Unsupervised Entity Alignment for Temporal Knowledge Graphs](https://mlsanity.com/paper/2302.00796/) (2023)
- [Ultra-Fine Entity Typing with Prior Knowledge about Labels: A Simple Clustering Based Strategy](https://mlsanity.com/paper/2305.12802/) (2023)
- [Recall, Expand and Multi-Candidate Cross-Encode: Fast and Accurate Ultra-Fine Entity Typing](https://mlsanity.com/paper/2212.09125/) (2022)
- [Universal Multi-modal Entity Alignment via Iteratively Fusing Modality Similarity Paths](https://mlsanity.com/paper/2310.05364/) (2023)

## Abstract (arXiv)

Ultra-fine entity typing (UFET) aims to predict a wide range of type phrases that correctly describe the categories of a given entity mention in a sentence. Most recent works infer each entity type independently, ignoring the correlations between types, e.g., when an entity is inferred as a president, it should also be a politician and a leader. To this end, we use an undirected graphical model called pairwise conditional random field (PCRF) to formulate the UFET problem, in which the type variables are not only unarily influenced by the input but also pairwisely relate to all the other type variables. We use various modern backbones for entity typing to compute unary potentials, and derive pairwise potentials from type phrase representations that both capture prior semantic information and facilitate accelerated inference. We use mean-field variational inference for efficient type inference on very large type sets and unfold it as a neural network module to enable end-to-end training. Experiments on UFET show that the Neural-PCRF consistently outperforms its backbones with little cost and results in a competitive performance against cross-encoder based SOTA while being thousands of times faster. We also find Neural- PCRF effective on a widely used fine-grained entity typing dataset with a smaller type set. We pack Neural-PCRF as a network module that can be plugged onto multi-label type classifiers with ease and release it in https://github.com/modelscope/adaseq/tree/master/examples/NPCRF.