# Larger-Context Tagging: When and Why Does It Work?

arXiv: https://arxiv.org/abs/2104.04434 · HTML: https://mlsanity.com/paper/2104.04434/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 47 references that are in the map.
- First posted on arXiv 9 Apr 2021.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [Neural Architectures for Named Entity Recognition](https://mlsanity.com/paper/1603.01360/) (2016)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF](https://mlsanity.com/paper/1603.01354/) (2016)
- [Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)](https://mlsanity.com/paper/2003.12206/) (2020)
- [Entity, Relation, and Event Extraction with Contextualized Span Representations](https://mlsanity.com/paper/1909.03546/) (2019)
- [Named Entity Recognition with Bidirectional LSTM-CNNs](https://mlsanity.com/paper/1511.08308/) (2015)
- [A Unified MRC Framework for Named Entity Recognition](https://mlsanity.com/paper/1910.11476/) (2019)
- [TENER: Adapting Transformer Encoder for Named Entity Recognition](https://mlsanity.com/paper/1911.04474/) (2019)
- [GraphIE: A Graph-Based Framework for Information Extraction](https://mlsanity.com/paper/1810.13083/) (2018)
- [Fast and Accurate Entity Recognition with Iterated Dilated Convolutions](https://mlsanity.com/paper/1702.02098/) (2017)
- [Rethinking Generalization of Neural Models: A Named Entity Recognition Case Study](https://mlsanity.com/paper/2001.03844/) (2020)
- [Hierarchical Contextualized Representation for Named Entity Recognition](https://mlsanity.com/paper/1911.02257/) (2019)
- [TriggerNER: Learning with Entity Triggers as Explanations for Named Entity Recognition](https://mlsanity.com/paper/2004.07493/) (2020)
- [Design Challenges and Misconceptions in Neural Sequence Labeling](https://mlsanity.com/paper/1806.04470/) (2018)
- [Interpretable Multi-dataset Evaluation for Named Entity Recognition](https://mlsanity.com/paper/2011.06854/) (2020)
- [Towards More Fine-grained and Reliable NLP Performance Prediction](https://mlsanity.com/paper/2102.05486/) (2021)
- [Instance-Based Learning of Span Representations: A Case Study through Named Entity Recognition](https://mlsanity.com/paper/2004.14514/) (2020)
- [Generalizing Natural Language Analysis through Span-relation Representations](https://mlsanity.com/paper/1911.03822/) (2019)
- [GRN: Gated Relation Network to Enhance Convolutional Neural Network for Named Entity Recognition](https://mlsanity.com/paper/1907.05611/) (2019)
- [Interpretability Analysis for Named Entity Recognition to Understand System Predictions and How They Can Improve](https://mlsanity.com/paper/2004.04564/) (2020)
- [Robust Lexical Features for Improved Neural Network Named-Entity Recognition](https://mlsanity.com/paper/1806.03489/) (2018)
- [RethinkCWS: Is Chinese Word Segmentation a Solved Task?](https://mlsanity.com/paper/2011.06858/) (2020)
- [Towards Accurate Word Segmentation for Chinese Patents](https://mlsanity.com/paper/1611.10038/) (2016)

## Built on by (most cited first, 4 of 4)

- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023)
- [SpanNER: Named Entity Re-/Recognition as Span Prediction](https://mlsanity.com/paper/2106.00641/) (2021)
- [What do we Really Know about State of the Art NER?](https://mlsanity.com/paper/2205.00034/) (2022)
- [Enhancing Label Consistency on Document-level Named Entity Recognition](https://mlsanity.com/paper/2210.12949/) (2022)

## Abstract (arXiv)

The development of neural networks and pretraining techniques has spawned many sentence-level tagging systems that achieved superior performance on typical benchmarks. However, a relatively less discussed topic is what if more context information is introduced into current top-scoring tagging systems. Although several existing works have attempted to shift tagging systems from sentence-level to document-level, there is still no consensus conclusion about when and why it works, which limits the applicability of the larger-context approach in tagging tasks. In this paper, instead of pursuing a state-of-the-art tagging system by architectural exploration, we focus on investigating when and why the larger-context training, as a general strategy, can work. To this end, we conduct a thorough comparative study on four proposed aggregators for context information collecting and present an attribute-aided evaluation method to interpret the improvement brought by larger-context training. Experimentally, we set up a testbed based on four tagging tasks and thirteen datasets. Hopefully, our preliminary observations can deepen the understanding of larger-context training and enlighten more follow-up works on the use of contextual information.