# Locally-Contextual Nonlinear CRFs for Sequence Labeling

arXiv: https://arxiv.org/abs/2103.16210 · HTML: https://mlsanity.com/paper/2103.16210/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 41 references that are in the map.
- First posted on arXiv 30 Mar 2021.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF](https://mlsanity.com/paper/1603.01354/) (2016)
- [LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](https://mlsanity.com/paper/2010.01057/) (2020)
- [A Unified MRC Framework for Named Entity Recognition](https://mlsanity.com/paper/1910.11476/) (2019)
- [Dice Loss for Data-imbalanced NLP Tasks](https://mlsanity.com/paper/1911.02855/) (2019)
- [Named Entity Recognition as Dependency Parsing](https://mlsanity.com/paper/2005.07150/) (2020)
- [Semi-Supervised Sequence Modeling with Cross-View Training](https://mlsanity.com/paper/1809.08370/) (2018)
- [Cloze-driven Pretraining of Self-attention Networks](https://mlsanity.com/paper/1903.07785/) (2019)
- [Hierarchical Contextualized Representation for Named Entity Recognition](https://mlsanity.com/paper/1911.02257/) (2019)
- [Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation](https://mlsanity.com/paper/1508.02096/) (2015)
- [Exploring Cross-sentence Contexts for Named Entity Recognition with BERT](https://mlsanity.com/paper/2006.01563/) (2020)
- [Sentence-State LSTM for Text Representation](https://mlsanity.com/paper/1805.02474/) (2018)
- [GCDT: A Global Context Enhanced Deep Transition Architecture for Sequence Labeling](https://mlsanity.com/paper/1906.02437/) (2019)
- [Morphosyntactic Tagging with a Meta-BiLSTM Model over Context Sensitive Token Encodings](https://mlsanity.com/paper/1805.08237/) (2018)
- [To BERT or Not to BERT: Comparing Task-specific and Task-agnostic Semi-Supervised Approaches for Sequence Tagging](https://mlsanity.com/paper/2010.14042/) (2020)
- [Heterogeneous Supervision for Relation Extraction: A Representation Learning Approach](https://mlsanity.com/paper/1707.00166/) (2017)
- [An improved neural network model for joint POS tagging and dependency parsing](https://mlsanity.com/paper/1807.03955/) (2018)
- [An Investigation of Potential Function Designs for Neural CRF](https://mlsanity.com/paper/2011.05604/) (2020)

## Built on by (most cited first, 2 of 2)

- [What do we Really Know about State of the Art NER?](https://mlsanity.com/paper/2205.00034/) (2022)
- [Improving VAE-based Representation Learning](https://mlsanity.com/paper/2205.14539/) (2022)

## Abstract (arXiv)

Linear chain conditional random fields (CRFs) combined with contextual word embeddings have achieved state of the art performance on sequence labeling tasks. In many of these tasks, the identity of the neighboring words is often the most useful contextual information when predicting the label of a given word. However, contextual embeddings are usually trained in a task-agnostic manner. This means that although they may encode information about the neighboring words, it is not guaranteed. It can therefore be beneficial to design the sequence labeling architecture to directly extract this information from the embeddings. We propose locally-contextual nonlinear CRFs for sequence labeling. Our approach directly incorporates information from the neighboring embeddings when predicting the label for a given word, and parametrizes the potential functions using deep neural networks. Our model serves as a drop-in replacement for the linear chain CRF, consistently outperforming it in our ablation study. On a variety of tasks, our results are competitive with those of the best published methods. In particular, we outperform the previous state of the art on chunking on CoNLL 2000 and named entity recognition on OntoNotes 5.0 English.