# Showing Your Work Doesn't Always Work

arXiv: https://arxiv.org/abs/2004.13705 · HTML: https://mlsanity.com/paper/2004.13705/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Apr 2020.

## Built on by (most cited first, 5 of 5)

- [Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters](https://mlsanity.com/paper/2110.04156/) (2021)
- [Expected Validation Performance and Estimation of a Random Variable's Maximum](https://mlsanity.com/paper/2110.00613/) (2021)
- [Transformers Go for the LOLs: Generating (Humourous) Titles from Scientific Abstracts End-to-End](https://mlsanity.com/paper/2212.10522/) (2022)
- [Show Your Work with Confidence: Confidence Bands for Tuning Curves](https://mlsanity.com/paper/2311.09480/) (2023)
- [Random Search Hyper-Parameter Tuning: Expected Improvement Estimation and the Corresponding Lower Bound](https://mlsanity.com/paper/2208.08170/) (2022)

## Abstract (arXiv)

In natural language processing, a recently popular line of work explores how to best report the experimental results of neural networks. One exemplar publication, titled "Show Your Work: Improved Reporting of Experimental Results," advocates for reporting the expected validation effectiveness of the best-tuned model, with respect to the computational budget. In the present work, we critically examine this paper. As far as statistical generalizability is concerned, we find unspoken pitfalls and caveats with this approach. We analytically show that their estimator is biased and uses error-prone assumptions. We find that the estimator favors negative errors and yields poor bootstrapped confidence intervals. We derive an unbiased alternative and bolster our claims with empirical evidence from statistical simulation. Our codebase is at http://github.com/castorini/meanmax.