# A Global Context Mechanism for Sequence Labeling

arXiv: https://arxiv.org/abs/2305.19928 · HTML: https://mlsanity.com/paper/2305.19928/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 52 references that are in the map.
- First posted on arXiv 31 May 2023.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF](https://mlsanity.com/paper/1603.01354/) (2016)
- [Sentiment Analysis in the Era of Large Language Models: A Reality Check](https://mlsanity.com/paper/2305.15005/) (2023)
- [Revisiting Pre-Trained Models for Chinese Natural Language Processing](https://mlsanity.com/paper/2004.13922/) (2020)
- [GPT-NER: Named Entity Recognition via Large Language Models](https://mlsanity.com/paper/2304.10428/) (2023)
- [Text Classification via Large Language Models](https://mlsanity.com/paper/2305.08377/) (2023)
- [A Unified Generative Framework for Aspect-Based Sentiment Analysis](https://mlsanity.com/paper/2106.04300/) (2021)
- [Interactive Attention Networks for Aspect-Level Sentiment Classification](https://mlsanity.com/paper/1709.00893/) (2017)
- [Exploiting BERT for End-to-End Aspect-based Sentiment Analysis](https://mlsanity.com/paper/1910.00883/) (2019)
- [A Unified Model for Opinion Target Extraction and Target Sentiment Prediction](https://mlsanity.com/paper/1811.05082/) (2018)
- [Neural Architectures for Nested NER through Linearization](https://mlsanity.com/paper/1908.06926/) (2019)
- [Hierarchical Contextualized Representation for Named Entity Recognition](https://mlsanity.com/paper/1911.02257/) (2019)
- [Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Loss](https://mlsanity.com/paper/1604.05529/) (2016)
- [Dependency-Guided LSTM-CRF for Named Entity Recognition](https://mlsanity.com/paper/1909.10148/) (2019)
- [Sentence-State LSTM for Text Representation](https://mlsanity.com/paper/1805.02474/) (2018)
- [GCDT: A Global Context Enhanced Deep Transition Architecture for Sequence Labeling](https://mlsanity.com/paper/1906.02437/) (2019)
- [Better Feature Integration for Named Entity Recognition](https://mlsanity.com/paper/2104.05316/) (2021)
- [Joint Aspect and Polarity Classification for Aspect-based Sentiment Analysis with End-to-End Neural Networks](https://mlsanity.com/paper/1808.09238/) (2018)
- [GRN: Gated Relation Network to Enhance Convolutional Neural Network for Named Entity Recognition](https://mlsanity.com/paper/1907.05611/) (2019)
- [DTMT: A Novel Deep Transition Architecture for Neural Machine Translation](https://mlsanity.com/paper/1812.07807/) (2018)
- [Robust Lexical Features for Improved Neural Network Named-Entity Recognition](https://mlsanity.com/paper/1806.03489/) (2018)

## Abstract (arXiv)

Global sentence information is crucial for sequence labeling tasks, where each word in a sentence must be assigned a label. While BiLSTM models are widely used, they often fail to capture sufficient global context for inner words. Previous work has proposed various RNN variants to integrate global sentence information into word representations. However, these approaches suffer from three key limitations: (1) they are slower in both inference and training compared to the original BiLSTM, (2) they cannot effectively supplement global information for transformer-based models, and (3) the high time cost associated with reimplementing and integrating these customized RNNs into existing architectures. In this study, we introduce a simple yet effective mechanism that addresses these limitations. Our approach efficiently supplements global sentence information for both BiLSTM and transformer-based models, with minimal degradation in inference and training speed, and is easily pluggable into current architectures. We demonstrate significant improvements in F1 scores across seven popular benchmarks, including Named Entity Recognition (NER) tasks such as Conll2003, Wnut2017 , and the Chinese named-entity recognition task Weibo, as well as End-to-End Aspect-Based Sentiment Analysis (E2E-ABSA) benchmarks such as Laptop14, Restaurant14, Restaurant15, and Restaurant16. With out any extra strategy, we achieve third highest score on weibo NER benchmark. Compared to CRF, one of the most popular frameworks for sequence labeling, our mechanism achieves competitive F1 scores while offering superior inference and training speed. Code is available at: https://github.com/conglei2XU/Global-Context-Mechanism