# SeqScore: Addressing Barriers to Reproducible Named Entity Recognition Evaluation

arXiv: https://arxiv.org/abs/2107.14154 · HTML: https://mlsanity.com/paper/2107.14154/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 42 references that are in the map.
- First posted on arXiv 29 Jul 2021.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](https://mlsanity.com/paper/2010.01057/) (2020)
- [A Unified MRC Framework for Named Entity Recognition](https://mlsanity.com/paper/1910.11476/) (2019)
- [MasakhaNER: Named Entity Recognition for African Languages](https://mlsanity.com/paper/2103.11811/) (2021)
- [Named Entity Recognition as Dependency Parsing](https://mlsanity.com/paper/2005.07150/) (2020)
- [Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning](https://mlsanity.com/paper/2105.03654/) (2021)
- [Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging](https://mlsanity.com/paper/1707.09861/) (2017)
- [Cloze-driven Pretraining of Self-attention Networks](https://mlsanity.com/paper/1903.07785/) (2019)
- [Design Challenges and Misconceptions in Neural Sequence Labeling](https://mlsanity.com/paper/1806.04470/) (2018)
- [Exploring Cross-sentence Contexts for Named Entity Recognition with BERT](https://mlsanity.com/paper/2006.01563/) (2020)
- [GCDT: A Global Context Enhanced Deep Transition Architecture for Sequence Labeling](https://mlsanity.com/paper/1906.02437/) (2019)
- [Constrained Decoding for Computationally Efficient Named Entity Recognition Taggers](https://mlsanity.com/paper/2010.04362/) (2020)
- [TMR: Evaluating NER Recall on Tough Mentions](https://mlsanity.com/paper/2103.12312/) (2021)
- [iobes: A Library for Span-Level Processing](https://mlsanity.com/paper/2010.04373/) (2020)

## Built on by (most cited first, 8 of 8)

- [Toward More Meaningful Resources for Lower-resourced Languages](https://mlsanity.com/paper/2202.12288/) (2022)
- [Investigating LLM Applications in E-Commerce](https://mlsanity.com/paper/2408.12779/) (2024)
- [Detecting Unassimilated Borrowings in Spanish: An Annotated Corpus and Approaches to Modeling](https://mlsanity.com/paper/2203.16169/) (2022)
- [QueryNER: Segmentation of E-commerce Queries](https://mlsanity.com/paper/2405.09507/) (2024)
- [Analysing Cross-Lingual Transfer in Low-Resourced African Named Entity Recognition](https://mlsanity.com/paper/2309.05311/) (2023)
- [CoNLL#: Fine-grained Error Analysis and a Corrected Test Set for CoNLL-03 English](https://mlsanity.com/paper/2405.11865/) (2024)
- [OpenNER 1.0: Standardized Open-Access Named Entity Recognition Datasets in 50+ Languages](https://mlsanity.com/paper/2412.09587/) (2024)
- [The GELATO Dataset for Legislative NER](https://mlsanity.com/paper/2603.14130/) (2026)

## Abstract (arXiv)

To address a looming crisis of unreproducible evaluation for named entity recognition, we propose guidelines and introduce SeqScore, a software package to improve reproducibility. The guidelines we propose are extremely simple and center around transparency regarding how chunks are encoded and scored. We demonstrate that despite the apparent simplicity of NER evaluation, unreported differences in the scoring procedure can result in changes to scores that are both of noticeable magnitude and statistically significant. We describe SeqScore, which addresses many of the issues that cause replication failures.