# Exploring Cross-sentence Contexts for Named Entity Recognition with BERT

arXiv: https://arxiv.org/abs/2006.01563 · HTML: https://mlsanity.com/paper/2006.01563/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jun 2020.

## Built on by (most cited first, 27 of 27)

- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning](https://mlsanity.com/paper/2105.03654/) (2021)
- [Packed Levitated Marker for Entity and Relation Extraction](https://mlsanity.com/paper/2109.06067/) (2021)
- [DAMO-NLP at SemEval-2022 Task 11: A Knowledge-based System for Multilingual Named Entity Recognition](https://mlsanity.com/paper/2203.00545/) (2022)
- [Context-NER : Contextual Phrase Generation at Scale](https://mlsanity.com/paper/2109.08079/) (2021)
- [Document-level Relation Extraction with Cross-sentence Reasoning Graph](https://mlsanity.com/paper/2303.03912/) (2023)
- [SeqScore: Addressing Barriers to Reproducible Named Entity Recognition Evaluation](https://mlsanity.com/paper/2107.14154/) (2021)
- [Partially Supervised Named Entity Recognition via the Expected Entity Ratio Loss](https://mlsanity.com/paper/2108.07216/) (2021)
- [CLIN-X: pre-trained language models and a study on cross-task transfer for concept extraction in the clinical domain](https://mlsanity.com/paper/2112.08754/) (2021)
- [WLV-RIT at SemEval-2021 Task 5: A Neural Transformer Framework for Detecting Toxic Spans](https://mlsanity.com/paper/2104.04630/) (2021)
- [Learning to Rank Context for Named Entity Recognition Using a Synthetic Dataset](https://mlsanity.com/paper/2310.10118/) (2023)
- [Locally-Contextual Nonlinear CRFs for Sequence Labeling](https://mlsanity.com/paper/2103.16210/) (2021)
- [The Role of Global and Local Context in Named Entity Recognition](https://mlsanity.com/paper/2305.03132/) (2023)
- [Transfer Learning for Multi-lingual Tasks -- a Survey](https://mlsanity.com/paper/2110.02052/) (2021)
- [Contextual Sentence Classification: Detecting Sustainability Initiatives in Company Reports](https://mlsanity.com/paper/2110.03727/) (2021)
- [Multilinguals at SemEval-2022 Task 11: Complex NER in Semantically Ambiguous Settings for Low Resource Languages](https://mlsanity.com/paper/2207.06882/) (2022)
- [Self-Adaptive Named Entity Recognition by Retrieving Unstructured Knowledge](https://mlsanity.com/paper/2210.07523/) (2022)
- [Syntactic Structure Processing in the Brain while Listening](https://mlsanity.com/paper/2302.08589/) (2023)
- [AlbNER: A Corpus for Named Entity Recognition in Albanian](https://mlsanity.com/paper/2309.08741/) (2023)
- [Transformer-based Named Entity Recognition with Combined Data Representation](https://mlsanity.com/paper/2406.17474/) (2024)
- [Optimizing small BERTs trained for German NER](https://mlsanity.com/paper/2104.11559/) (2021)
- [KARL-Trans-NER: Knowledge Aware Representation Learning for Named Entity Recognition using Transformers](https://mlsanity.com/paper/2111.15436/) (2021)
- [GlobalNER: Incorporating Non-local Information into Named Entity Recognition](https://mlsanity.com/paper/2303.02915/) (2023)
- [Fine Tuning Named Entity Extraction Models for the Fantasy Domain](https://mlsanity.com/paper/2402.10662/) (2024)
- [Academic case reports lack diversity: Assessing the presence and diversity of sociodemographic and behavioral factors related to Post COVID-19 Condition](https://mlsanity.com/paper/2501.12538/) (2025)
- [Nested Named-Entity Recognition on Vietnamese COVID-19: Dataset and Experiments](https://mlsanity.com/paper/2504.21016/) (2025)
- [Comparing BERT Sentence-Pair Classification and Few-Shot LLM Prompting for Detecting Threat and Solution Framing in German Climate News](https://mlsanity.com/paper/2606.26489/) (2026)

## Abstract (arXiv)

Named entity recognition (NER) is frequently addressed as a sequence classification task where each input consists of one sentence of text. It is nevertheless clear that useful information for the task can often be found outside of the scope of a single-sentence context. Recently proposed self-attention models such as BERT can both efficiently capture long-distance relationships in input as well as represent inputs consisting of several sentences, creating new opportunitites for approaches that incorporate cross-sentence information in natural language processing tasks. In this paper, we present a systematic study exploring the use of cross-sentence information for NER using BERT models in five languages. We find that adding context in the form of additional sentences to BERT input systematically increases NER performance on all of the tested languages and models. Including multiple sentences in each input also allows us to study the predictions of the same sentences in different contexts. We propose a straightforward method, Contextual Majority Voting (CMV), to combine different predictions for sentences and demonstrate this to further increase NER performance with BERT. Our approach does not require any changes to the underlying BERT architecture, rather relying on restructuring examples for training and prediction. Evaluation on established datasets, including the CoNLL'02 and CoNLL'03 NER benchmarks, demonstrates that our proposed approach can improve on the state-of-the-art NER results on English, Dutch, and Finnish, achieves the best reported BERT-based results on German, and is on par with performance reported with other BERT-based approaches in Spanish. We release all methods implemented in this work under open licenses.