mlsanityData to 14 Oct 2026

SeqScore: Addressing Barriers to Reproducible Named Entity Recognition Evaluation

arXiv 2107.14154 ↗First posted 29 Jul 2021cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

To address a looming crisis of unreproducible evaluation for named entity recognition, we propose guidelines and introduce SeqScore, a software package to improve reproducibility. The guidelines we propose are extremely simple and center around transparency regarding how chunks are encoded and scored. We demonstrate that despite the apparent simplicity of NER evaluation, unreported differences in the scoring procedure can result in changes to scores that are both of noticeable magnitude and statistically significant. We describe SeqScore, which addresses many of the issues that cause replication failures.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 17 of its 42 references that are in the map. First posted on arXiv 29 Jul 2021.

Builds on

17 of 42 references.

References · 17

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  3. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  4. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 723
  5. 2020LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attentioncited by 326
  6. 2019A Unified MRC Framework for Named Entity Recognitioncited by 199
  7. 2021MasakhaNER: Named Entity Recognition for African Languagescited by 155
  8. 2020Named Entity Recognition as Dependency Parsingcited by 151
  9. 2021Improving Named Entity Recognition by External Context Retrieving and Cooperative Learningcited by 76
  10. 2017Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Taggingcited by 73
  11. 2019Cloze-driven Pretraining of Self-attention Networkscited by 41
  12. 2018Design Challenges and Misconceptions in Neural Sequence Labelingcited by 28
5 more
  1. 2020Exploring Cross-sentence Contexts for Named Entity Recognition with BERTcited by 27
  2. 2019GCDT: A Global Context Enhanced Deep Transition Architecture for Sequence Labelingcited by 22
  3. 2020Constrained Decoding for Computationally Efficient Named Entity Recognition Taggerscited by 8
  4. 2021TMR: Evaluating NER Recall on Tough Mentionscited by 4
  5. 2020iobes: A Library for Span-Level Processingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 1; 2024: 4; 2025: 0; 2026: 1.

8 papers.

  1. 2022Toward More Meaningful Resources for Lower-resourced Languagescited by 14
  2. 2024Investigating LLM Applications in E-Commercecited by 7
  3. 2022Detecting Unassimilated Borrowings in Spanish: An Annotated Corpus and Approaches to Modelingcited by 6
  4. 2024QueryNER: Segmentation of E-commerce Queriescited by 3
  5. 2023Analysing Cross-Lingual Transfer in Low-Resourced African Named Entity Recognitioncited by 2
  6. 2024CoNLL#: Fine-grained Error Analysis and a Corrected Test Set for CoNLL-03 Englishcited by 2
  7. 2024OpenNER 1.0: Standardized Open-Access Named Entity Recognition Datasets in 50+ Languagescited by 2
  8. 2026The GELATO Dataset for Legislative NERcited by 0