# Expected Validation Performance and Estimation of a Random Variable's Maximum

arXiv: https://arxiv.org/abs/2110.00613 · HTML: https://mlsanity.com/paper/2110.00613/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 17 references that are in the map.
- First posted on arXiv 1 Oct 2021.

## Builds on (most cited first, 13 of 13)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Underspecification Presents Challenges for Credibility in Modern Machine Learning](https://mlsanity.com/paper/2011.03395/) (2020)
- [Revisiting Few-sample BERT Fine-tuning](https://mlsanity.com/paper/2006.05987/) (2020)
- [Show Your Work: Improved Reporting of Experimental Results](https://mlsanity.com/paper/1909.03004/) (2019)
- [On the State of the Art of Evaluation in Neural Language Models](https://mlsanity.com/paper/1707.05589/) (2017)
- [Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging](https://mlsanity.com/paper/1707.09861/) (2017)
- [Utility is in the Eye of the User: A Critique of NLP Leaderboards](https://mlsanity.com/paper/2009.13888/) (2020)
- [The Scientific Method in the Science of Machine Learning](https://mlsanity.com/paper/1904.10922/) (2019)
- [Bayesian Optimization of Text Representations](https://mlsanity.com/paper/1503.00693/) (2015)
- [Showing Your Work Doesn't Always Work](https://mlsanity.com/paper/2004.13705/) (2020)

## Built on by (most cited first, 6 of 6)

- [Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters](https://mlsanity.com/paper/2110.04156/) (2021)
- [HyperMixer: An MLP-based Low Cost Alternative to Transformers](https://mlsanity.com/paper/2203.03691/) (2022)
- [Linear Transformers with Learnable Kernel Functions are Better In-Context Models](https://mlsanity.com/paper/2402.10644/) (2024)
- [Show Your Work with Confidence: Confidence Bands for Tuning Curves](https://mlsanity.com/paper/2311.09480/) (2023)
- [Stronger Random Baselines for In-Context Learning](https://mlsanity.com/paper/2404.13020/) (2024)
- [Random Search Hyper-Parameter Tuning: Expected Improvement Estimation and the Corresponding Lower Bound](https://mlsanity.com/paper/2208.08170/) (2022)

## Abstract (arXiv)

Research in NLP is often supported by experimental results, and improved reporting of such results can lead to better understanding and more reproducible science. In this paper we analyze three statistical estimators for expected validation performance, a tool used for reporting performance (e.g., accuracy) as a function of computational budget (e.g., number of hyperparameter tuning experiments). Where previous work analyzing such estimators focused on the bias, we also examine the variance and mean squared error (MSE). In both synthetic and realistic scenarios, we evaluate three estimators and find the unbiased estimator has the highest variance, and the estimator with the smallest variance has the largest bias; the estimator with the smallest MSE strikes a balance between bias and variance, displaying a classic bias-variance tradeoff. We use expected validation performance to compare between different models, and analyze how frequently each estimator leads to drawing incorrect conclusions about which of two models performs best. We find that the two biased estimators lead to the fewest incorrect conclusions, which hints at the importance of minimizing variance and MSE.