mlsanityData to 7 Oct 2026

Show Your Work: Improved Reporting of Experimental Results

arXiv 1909.03004 ↗First posted 6 Sep 2019cs.LG cs.CL stat.MEcited by 164

Not on the map. Why?

Abstract (arXiv)

Research in natural language processing proceeds, in part, by demonstrating that new models achieve superior performance (e.g., accuracy) on held-out test data, compared to previous results. In this paper, we demonstrate that test-set performance scores alone are insufficient for drawing accurate conclusions about which model performs best. We argue for reporting additional details, especially performance on validation data obtained during model development. We present a novel technique for doing so: expected validation performance of the best-found model as a function of computation budget (i.e., the number of hyperparameter search trials or the overall training time). Using our approach, we find multiple recent model comparisons where authors would have reached a different conclusion if they had used more (or less) computation. Our approach also allows us to estimate the amount of computation required to obtain a given accuracy; applying it to several recently published results yields massive variation across papers, from hours to weeks. We conclude with a set of best practices for reporting experimental results which allow for robust future comparisons, and provide code to allow researchers to use our technique.

Key facts. Cited by 164 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 43; 2022: 28; 2023: 22; 2024: 14; 2025: 16; 2026: 41.

164 papers.

Sort:

By area:

  1. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  2. 2021True Few-Shot Learning with Language Modelscited by 343
  3. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 340
  4. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  5. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 200
  6. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  7. 2021The Benchmark Lotterycited by 133
  8. 2021Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Researchcited by 100
  9. 2021UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmarkcited by 98
  10. 2022Design-Bench: Benchmarks for Data-Driven Offline Model-Based Optimizationcited by 96
  11. 2024Better than classical? The subtle art of benchmarking quantum machine learning modelscited by 82
  12. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
38 more
  1. 2021Anticipating Safety Issues in E2E Conversational AI: Framework and Toolingcited by 77
  2. 2021Is Automated Topic Model Evaluation Broken?: The Incoherence of Coherencecited by 74
  3. 2021Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Wordscited by 63
  4. 2021Variational Information Bottleneck for Effective Low-Resource Fine-Tuningcited by 51
  5. 2022Towards a Standardised Performance Evaluation Protocol for Cooperative MARLcited by 48
  6. 2021Just What do You Think You're Doing, Dave?' A Checklist for Responsible Data Use in NLPcited by 47
  7. 2021Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translationcited by 44
  8. 2024Can Unconfident LLM Annotations Be Used for Confident Conclusions?cited by 44
  9. 2021The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Failcited by 38
  10. 2021Multilingual and Cross-Lingual Intent Detection from Spoken Datacited by 35
  11. 2022Data Governance in the Age of Large-Scale Data-Driven Language Technologycited by 32
  12. 2021Exploring the Role of BERT Token Representations to Explain Sentence Probing Resultscited by 29
  13. 2021Learning with Different Amounts of Annotation: From Zero to Many Labelscited by 28
  14. 2022deep-significance - Easy and Meaningful Statistical Significance Testing in the Age of Neural Networkscited by 28
  15. 2022Sources of Irreproducibility in Machine Learning: A Reviewcited by 28
  16. 2024AnthroScore: A Computational Linguistic Measure of Anthropomorphismcited by 27
  17. 2021Towards More Fine-grained and Reliable NLP Performance Predictioncited by 26
  18. 2023Reflection-Tuning: Data Recycling Improves LLM Instruction-Tuningcited by 26
  19. 2022FairGBM: Gradient Boosting with Fairness Constraintscited by 25
  20. 2021Preregistering NLP Researchcited by 24
  21. 2021How Robust are Model Rankings: A Leaderboard Customization Approach for Equitable Evaluationcited by 24
  22. 2021Reusable Templates and Guides For Documenting Datasets and Models for Natural Language Processing and Generation: A Case Study of the HuggingFace and GEM Data and Model Cardscited by 24
  23. 2021Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matterscited by 24
  24. 2024What Can Natural Language Processing Do for Peer Review?cited by 24
  25. 2021Convex Aggregation for Opinion Summarizationcited by 21
  26. 2022Great Power, Great Responsibility: Recommendations for Reducing Energy for Training Language Modelscited by 21
  27. 2023Concrete Problems in AI Safety, Revisitedcited by 21
  28. 2021Hyperparameter Optimization Is Deceiving Us, and How to Stop Itcited by 20
  29. 2023Position: Key Claims in LLM Research Have a Long Tail of Footnotescited by 20
  30. 2021Underreporting of errors in NLG output, and what to do about itcited by 19
  31. 2021Comparative Opinion Summarization via Collaborative Decodingcited by 18
  32. 2023Reproducibility in NLP: What Have We Learned from the Checklist?cited by 18
  33. 2025Reality Check: A New Evaluation Ecosystem Is Necessary to Understand AI's Real World Effectscited by 18
  34. 2022The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learningcited by 17
  35. 2021Narrative Question Answering with Cutting-Edge Open-Domain QA Techniques: A Comprehensive Studycited by 15
  36. 2025Don't Pass@k: A Bayesian Framework for Large Language Model Evaluationcited by 15
  37. 2023When Good and Reproducible Results are a Giant with Feet of Clay: The Importance of Software Quality in NLPcited by 14
  38. 2026SafetyRepro: Configuration-Conditional Rank Instability on Alignment Benchmarkscited by 14

50 most cited of 164 All 164 (JSON)