mlsanityData to 7 Oct 2026

With Little Power Comes Great Responsibility

arXiv 2010.06595 ↗First posted 13 Oct 2020cs.CL cs.AI cs.LGcited by 138

Not on the map. Why?

Abstract (arXiv)

Despite its importance to experimental design, statistical power (the probability that, given a real effect, an experiment will reject the null hypothesis) has largely been ignored by the NLP community. Underpowered experiments make it more difficult to discern the difference between statistical noise and meaningful model improvements, and increase the chances of exaggerated findings. By meta-analyzing a set of existing NLP papers and datasets, we characterize typical power for a variety of settings and conclude that underpowered experiments are common in the NLP literature. In particular, for several tasks in the popular GLUE benchmark, small test sets mean that most attempted comparisons to state of the art models will not be adequately powered. Similarly, based on reasonable assumptions, we find that the most typical experimental design for human rating studies will be underpowered to detect small model differences, of the sort that are frequently studied. For machine translation, we find that typical test sets of 2000 sentences have approximately 75% power to detect differences of 1 BLEU point. To improve the situation going forward, we give an overview of best practices for power analysis in NLP and release a series of notebooks to assist with future power analyses.

Key facts. Cited by 138 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 25; 2022: 24; 2023: 13; 2024: 16; 2025: 17; 2026: 43.

138 papers.

Sort:

By area:

  1. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  2. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  3. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  4. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  5. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 200
  6. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  7. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  8. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  9. 2021The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generationcited by 96
  10. 2023Guiding Large Language Models via Directional Stimulus Promptingcited by 94
  11. 2023Document-Level Machine Translation with Large Language Modelscited by 90
  12. 2021The MultiBERTs: BERT Reproductions for Robustness Analysiscited by 89
38 more
  1. 2021Is GPT-3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Textcited by 89
  2. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 89
  3. 2023Anchor Points: Benchmarking Models with Much Fewer Examplescited by 84
  4. 2021Is Automated Topic Model Evaluation Broken?: The Incoherence of Coherencecited by 74
  5. 2022A fine-grained comparison of pragmatic language understanding in humans and language modelscited by 69
  6. 2021FLEX: Unifying Evaluation for Few-Shot NLPcited by 67
  7. 2021Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarkingcited by 53
  8. 2021AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languagescited by 50
  9. 2021The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translationcited by 46
  10. 2024Can Unconfident LLM Annotations Be Used for Confident Conclusions?cited by 44
  11. 2021Automatic Text Evaluation through the Lens of Wasserstein Barycenterscited by 38
  12. 2021The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Failcited by 38
  13. 2022How Large Language Models are Transforming Machine-Paraphrased Plagiarismcited by 38
  14. 2024Suri: Multi-constraint Instruction Following for Long-form Text Generationcited by 34
  15. 2022deep-significance - Easy and Meaningful Statistical Significance Testing in the Age of Neural Networkscited by 28
  16. 2022Sources of Irreproducibility in Machine Learning: A Reviewcited by 28
  17. 2024SPINACH: SPARQL-Based Information Navigation for Challenging Real-World Questionscited by 27
  18. 2022The Authenticity Gap in Human Evaluationcited by 24
  19. 2024Annotation alignment: Comparing LLM and human annotations of conversational safetycited by 22
  20. 2025Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluationcited by 22
  21. 2021Do Question Answering Modeling Improvements Hold Across Benchmarks?cited by 20
  22. 2025Fairness through Difference Awareness: Measuring Desired Group Discrimination in LLMscited by 20
  23. 2021The Human Evaluation Datasheet 1.0: A Template for Recording Details of Human Evaluation Experiments in NLPcited by 19
  24. 2021Underreporting of errors in NLG output, and what to do about itcited by 19
  25. 2022On the Limitations of Dataset Balancing: The Lost Battle Against Spurious Correlationscited by 19
  26. 2021The statistical advantage of automatic NLG metrics at the system levelcited by 18
  27. 2024The Challenges of Evaluating LLM Applications: An Analysis of Automated, Human, and LLM-Based Approachescited by 18
  28. 2021Controllable Summarization with Constrained Markov Decision Processcited by 17
  29. 2022The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learningcited by 17
  30. 2022Multi-CLS BERT: An Efficient Alternative to Traditional Ensemblingcited by 14
  31. 2025Potemkin Understanding in Large Language Modelscited by 14
  32. 2023A Call for Standardization and Validation of Text Style Transfer Evaluationcited by 13
  33. 2021A Search Engine for Discovery of Scientific Challenges and Directionscited by 12
  34. 2023Pre-registration for Predictive Modelingcited by 11
  35. 2025PCoT: Persuasion-Augmented Chain of Thought for Detecting Fake News and Social Media Disinformationcited by 11
  36. 2025Is GPT-OSS Good? A Comprehensive Evaluation of OpenAI's Latest Open Source Modelscited by 11
  37. 2022Textwash -- automated open-source text anonymisationcited by 10
  38. 2022Needle in a Haystack: An Analysis of High-Agreement Workers on MTurk for Summarizationcited by 10

50 most cited of 138 All 138 (JSON)