# Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging

arXiv: https://arxiv.org/abs/1707.09861 · HTML: https://mlsanity.com/paper/1707.09861/

## Key facts

- Cited by 73 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Jul 2017.

## Built on by (most cited first, 30 of 73)

- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Accounting for Variance in Machine Learning Benchmarks](https://mlsanity.com/paper/2103.03098/) (2021)
- [Efficient Methods for Natural Language Processing: A Survey](https://mlsanity.com/paper/2209.00099/) (2022)
- [Voting or Consensus? Decision-Making in Multi-Agent Debate](https://mlsanity.com/paper/2502.19130/) (2025)
- [Temporal Effects on Pre-trained Models for Language Processing Tasks](https://mlsanity.com/paper/2111.12790/) (2021)
- [Falsesum: Generating Document-level NLI Examples for Recognizing Factual Inconsistency in Summarization](https://mlsanity.com/paper/2205.06009/) (2022)
- [Sources of Irreproducibility in Machine Learning: A Review](https://mlsanity.com/paper/2204.07610/) (2022)
- [CySecBERT: A Domain-Adapted Language Model for the Cybersecurity Domain](https://mlsanity.com/paper/2212.02974/) (2022)
- [How Robust are Model Rankings: A Leaderboard Customization Approach for Equitable Evaluation](https://mlsanity.com/paper/2106.05532/) (2021)
- [How much pretraining data do language models need to learn syntax?](https://mlsanity.com/paper/2109.03160/) (2021)
- [Fair and Argumentative Language Modeling for Computational Argumentation](https://mlsanity.com/paper/2204.04026/) (2022)
- [Time for aCTIon: Automated Analysis of Cyber Threat Intelligence in the Wild](https://mlsanity.com/paper/2307.10214/) (2023)
- [A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomness](https://mlsanity.com/paper/2312.01082/) (2023)
- [We need to talk about random seeds](https://mlsanity.com/paper/2210.13393/) (2022)
- [Few-shot clinical entity recognition in English, French and Spanish: masked language models outperform generative model prompting](https://mlsanity.com/paper/2402.12801/) (2024)
- [Cross-Domain Multi-Task Learning for Sequential Sentence Classification in Research Papers](https://mlsanity.com/paper/2102.06008/) (2021)
- [Modeling the Machine Learning Multiverse](https://mlsanity.com/paper/2206.05985/) (2022)
- [Same Scores, Different Decisions: Evaluating JEV and Language Models for Legal Document Understanding](https://mlsanity.com/paper/2609.27678/) (2026)
- [Innovations in Neural Data-to-text Generation: A Survey](https://mlsanity.com/paper/2207.12571/) (2022)
- [Perspectives on Machine Learning from Psychology's Reproducibility Crisis](https://mlsanity.com/paper/2104.08878/) (2021)
- [Do CoNLL-2003 Named Entity Taggers Still Work Well in 2023?](https://mlsanity.com/paper/2212.09747/) (2022)
- [Metrical Tagging in the Wild: Building and Annotating Poetry Corpora with Rhythmic Features](https://mlsanity.com/paper/2102.08858/) (2021)
- [SeqScore: Addressing Barriers to Reproducible Named Entity Recognition Evaluation](https://mlsanity.com/paper/2107.14154/) (2021)
- [Not All Linearizations Are Equally Data-Hungry in Sequence Labeling Parsing](https://mlsanity.com/paper/2108.07556/) (2021)
- [Introducing Rhetorical Parallelism Detection: A New Task with Datasets, Metrics, and Baselines](https://mlsanity.com/paper/2312.00100/) (2023)
- [When +1% Is Not Enough: A Paired Bootstrap Protocol for Evaluating Small Improvements](https://mlsanity.com/paper/2511.19794/) (2025)
- [Multi-Task Attentive Residual Networks for Argument Mining](https://mlsanity.com/paper/2102.12227/) (2021)
- [Learning to Bridge Metric Spaces: Few-shot Joint Learning of Intent Detection and Slot Filling](https://mlsanity.com/paper/2106.07343/) (2021)
- [Expected Validation Performance and Estimation of a Random Variable's Maximum](https://mlsanity.com/paper/2110.00613/) (2021)
- [Can Unsupervised Knowledge Transfer from Social Discussions Help Argument Mining?](https://mlsanity.com/paper/2203.12881/) (2022)

## Abstract (arXiv)

In this paper we show that reporting a single performance score is insufficient to compare non-deterministic approaches. We demonstrate for common sequence tagging tasks that the seed value for the random number generator can result in statistically significant (p < 10^-4) differences for state-of-the-art systems. For two recent systems for NER, we observe an absolute difference of one percentage point F1-score depending on the selected seed value, making these systems perceived either as state-of-the-art or mediocre. Instead of publishing and reporting single performance scores, we propose to compare score distributions based on multiple executions. Based on the evaluation of 50.000 LSTM-networks for five sequence tagging tasks, we present network architectures that produce both superior performance as well as are more stable with respect to the remaining hyperparameters.