Beyond statistical significance: Quantifying uncertainty and statistical variability in multilingual and multitask NLP evaluation
Not on the map. Why?
Abstract (arXiv)
We introduce a set of resampling-based methods for quantifying uncertainty and statistical precision of evaluation metrics in multilingual and/or multitask NLP benchmarks. We show how experimental variation in performance scores arises from both model and data-related sources, and that accounting for both of them is necessary to avoid substantially underestimating the overall variability over hypothetical replications. Using multilingual question answering, machine translation, and named entity recognition as example tasks, we also demonstrate how resampling methods are useful for quantifying the replication uncertainty of various quantities used in leaderboards such as model rankings and pairwise differences between models.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 27 of its 48 references that are in the map. First posted on arXiv 26 Sep 2025.
Builds on
27 of 48 references.
References · 27
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
- 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
- 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
- 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
- 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 206
- 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 163
- 2024Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similaritiescited by 159
- 2021The Benchmark Lotterycited by 136
- 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
15 more
- 2017Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Taggingcited by 73
- 2020We Need to Talk About Random Splitscited by 71
- 2024MAQA: Evaluating Uncertainty Quantification in LLMs Regarding Data Uncertaintycited by 27
- 2024Towards Reproducible LLM Evaluation: Quantifying Uncertainty in LLM Benchmark Scorescited by 25
- 2025Understanding the Uncertainty of LLM Explanations: A Perspective Based on Reasoning Topologycited by 25
- 2024Black-box Uncertainty Quantification Method for LLM-as-a-Judgecited by 18
- 2025Uncertainty Quantification of Large Language Models through Multi-Dimensional Responsescited by 18
- 2022We need to talk about random seedscited by 13
- 2022Experimental Standards for Deep Learning in Natural Language Processing Researchcited by 10
- 2025Statistical Uncertainty Quantification for Aggregate Performance Metrics in Machine Learning Benchmarkscited by 10
- 2021How not to Lie with a Benchmark: Rearranging NLP Leaderboardscited by 9
- 2022Investigating Data Variance in Evaluations of Automatic Machine Translation Metricscited by 3
- 2024Question Rephrasing for Quantifying Uncertainty in Large Language Models: Applications in Molecular Chemistry Taskscited by 3
- 2024OpenNER 1.0: Standardized Open-Access Named Entity Recognition Datasets in 50+ Languagescited by 2
- 2024The Effect of Data Partitioning Strategy on Model Generalizability: A Case Study of Morphological Segmentationcited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.