# Why Attention? Analyzing and Remedying BiLSTM Deficiency in Modeling Cross-Context for NER

arXiv: https://arxiv.org/abs/1910.02586 · HTML: https://mlsanity.com/paper/1910.02586/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Oct 2019.

## Built on by (most cited first, 2 of 2)

- [Context-NER : Contextual Phrase Generation at Scale](https://mlsanity.com/paper/2109.08079/) (2021)
- [Enhancing Speech Instruction Understanding and Disambiguation in Robotics via Speech Prosody](https://mlsanity.com/paper/2506.02057/) (2025)

## Abstract (arXiv)

State-of-the-art approaches of NER have used sequence-labeling BiLSTM as a core module. This paper formally shows the limitation of BiLSTM in modeling cross-context patterns. Two types of simple cross-structures -- self-attention and Cross-BiLSTM -- are shown to effectively remedy the problem. On both OntoNotes 5.0 and WNUT 2017, clear and consistent improvements are achieved over bare-bone models, up to 8.7% on some of the multi-token mentions. In-depth analyses across several aspects of the improvements, especially the identification of multi-token mentions, are further given.