# Climbing a shaky ladder: Better adaptive risk estimation

arXiv: https://arxiv.org/abs/1706.02733 · HTML: https://mlsanity.com/paper/1706.02733/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jun 2017.

## Built on by (most cited first, 6 of 6)

- [Overtuning in Hyperparameter Optimization](https://mlsanity.com/paper/2506.19540/) (2025)
- [LLM Personas as a Substitute for Field Experiments in Method Benchmarking](https://mlsanity.com/paper/2512.21080/) (2025)
- [Responsible AI in Open Ecosystems: Reconciling Innovation with Risk Assessment and Disclosure](https://mlsanity.com/paper/2409.19104/) (2024)
- [ShakyPrepend: A Multi-Group Learner with Improved Sample Complexity](https://mlsanity.com/paper/2603.07319/) (2026)
- [How Sensitive Are LLM Leaderboard Claims to Hidden Model Selection?](https://mlsanity.com/paper/2609.28177/) (2026)
- [How Reusable Are Benchmarks with Richer Feedback?](https://mlsanity.com/paper/2609.32109/) (2026)

## Abstract (arXiv)

We revisit the leaderboard problem introduced by Blum and Hardt (2015) in an effort to reduce overfitting in machine learning benchmarks. We show that a randomized version of their Ladder algorithm achieves leaderboard error O(1/n0.4) compared with the previous best rate of O(1/n1/3). Short of proving that our algorithm is optimal, we point out a major obstacle toward further progress. Specifically, any improvement to our upper bound would lead to asymptotic improvements in the general adaptive estimation setting as have remained elusive in recent years. This connection also directly leads to lower bounds for specific classes of algorithms. In particular, we exhibit a new attack on the leaderboard algorithm that both theoretically and empirically distinguishes between our algorithm and previous leaderboard algorithms.