mlsanityData to 14 Oct 2026

Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging

arXiv 1707.09861 ↗First posted 31 Jul 2017cs.CL stat.MLcited by 73

Not on the map. Why?

Abstract (arXiv)

In this paper we show that reporting a single performance score is insufficient to compare non-deterministic approaches. We demonstrate for common sequence tagging tasks that the seed value for the random number generator can result in statistically significant (p < 10^-4) differences for state-of-the-art systems. For two recent systems for NER, we observe an absolute difference of one percentage point F1-score depending on the selected seed value, making these systems perceived either as state-of-the-art or mediocre. Instead of publishing and reporting single performance scores, we propose to compare score distributions based on multiple executions. Based on the evaluation of 50.000 LSTM-networks for five sequence tagging tasks, we present network architectures that produce both superior performance as well as are more stable with respect to the remaining hyperparameters.

Key facts. Cited by 73 papers in the map (data through 14 Oct 2026). First posted on arXiv 31 Jul 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 18; 2022: 14; 2023: 10; 2024: 4; 2025: 9; 2026: 18.

73 papers.

Sort:

By area:

  1. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  2. 2021Accounting for Variance in Machine Learning Benchmarkscited by 201
  3. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  4. 2025Voting or Consensus? Decision-Making in Multi-Agent Debatecited by 59
  5. 2021Temporal Effects on Pre-trained Models for Language Processing Taskscited by 46
  6. 2022Falsesum: Generating Document-level NLI Examples for Recognizing Factual Inconsistency in Summarizationcited by 34
  7. 2022Sources of Irreproducibility in Machine Learning: A Reviewcited by 32
  8. 2022CySecBERT: A Domain-Adapted Language Model for the Cybersecurity Domaincited by 31
  9. 2021How Robust are Model Rankings: A Leaderboard Customization Approach for Equitable Evaluationcited by 24
  10. 2021How much pretraining data do language models need to learn syntax?cited by 21
  11. 2022Fair and Argumentative Language Modeling for Computational Argumentationcited by 18
  12. 2023Time for aCTIon: Automated Analysis of Cyber Threat Intelligence in the Wildcited by 15
38 more
  1. 2023A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomnesscited by 14
  2. 2022We need to talk about random seedscited by 13
  3. 2024Few-shot clinical entity recognition in English, French and Spanish: masked language models outperform generative model promptingcited by 12
  4. 2021Cross-Domain Multi-Task Learning for Sequential Sentence Classification in Research Paperscited by 11
  5. 2022Modeling the Machine Learning Multiversecited by 11
  6. 2026Same Scores, Different Decisions: Evaluating JEV and Language Models for Legal Document Understandingcited by 11
  7. 2022Innovations in Neural Data-to-text Generation: A Surveycited by 10
  8. 2021Perspectives on Machine Learning from Psychology's Reproducibility Crisiscited by 9
  9. 2022Do CoNLL-2003 Named Entity Taggers Still Work Well in 2023?cited by 9
  10. 2021Metrical Tagging in the Wild: Building and Annotating Poetry Corpora with Rhythmic Featurescited by 8
  11. 2021SeqScore: Addressing Barriers to Reproducible Named Entity Recognition Evaluationcited by 8
  12. 2021Not All Linearizations Are Equally Data-Hungry in Sequence Labeling Parsingcited by 7
  13. 2023Introducing Rhetorical Parallelism Detection: A New Task with Datasets, Metrics, and Baselinescited by 7
  14. 2025When +1% Is Not Enough: A Paired Bootstrap Protocol for Evaluating Small Improvementscited by 7
  15. 2021Multi-Task Attentive Residual Networks for Argument Miningcited by 6
  16. 2021Learning to Bridge Metric Spaces: Few-shot Joint Learning of Intent Detection and Slot Fillingcited by 6
  17. 2021Expected Validation Performance and Estimation of a Random Variable's Maximumcited by 6
  18. 2022Can Unsupervised Knowledge Transfer from Social Discussions Help Argument Mining?cited by 5
  19. 2022Bridging Fairness and Environmental Sustainability in Natural Language Processingcited by 5
  20. 2022Did AI get more negative recently?cited by 4
  21. 2023Towards Inferential Reproducibility of Machine Learning Researchcited by 4
  22. 2023How to Handle Different Types of Out-of-Distribution Scenarios in Computational Argumentation? A Comprehensive and Fine-Grained Field Studycited by 4
  23. 2025The Value of Nothing: Multimodal Extraction of Human Values Expressed by TikTok Influencerscited by 4
  24. 2026Are Sparse Autoencoder Benchmarks Reliable?cited by 4
  25. 2021Challenges for cognitive decoding using deep learning methodscited by 3
  26. 2021Evaluating deep transfer learning for whole-brain cognitive decodingcited by 3
  27. 2025Evaluation of Seismic Artificial Intelligence with Uncertaintycited by 3
  28. 2025PTEB: Towards Robust Text Embedding Evaluation via Stochastic Paraphrasing at Evaluation Time with LLMscited by 3
  29. 2022Data-driven Model Generalizability in Crosslinguistic Low-resource Morphological Segmentationcited by 2
  30. 2022Parsimonious Argument Annotations for Hate Speech Counter-narrativescited by 2
  31. 2023CebuaNER: A New Baseline Cebuano Named Entity Recognition Modelcited by 2
  32. 2023Decoupling Representation and Knowledge for Few-Shot Intent Classification and Slot Fillingcited by 2
  33. 2024ASTE Transformer Modelling Dependencies in Aspect-Sentiment Triplet Extractioncited by 2
  34. 2025Not quite Sherlock Holmes: Language model predictions do not reliably differentiate impossible from improbable eventscited by 2
  35. 2021Investigation on Data Adaptation Techniques for Neural Named Entity Recognitioncited by 1
  36. 2024On Sensitivity of Learning with Limited Labelled Data to the Effects of Randomness: Impact of Interactions and Systematic Choicescited by 1
  37. 2024Measuring training variability from stochastic optimization using robust nonparametric testingcited by 1
  38. 2025Comparative Study of Zero-Shot Cross-Lingual Transfer for Bodo POS and NER Tagging Using Gemini 2.0 Flash Thinking Experimental Modelcited by 1

50 most cited of 73 All 73 (JSON)