# Minimizers of the Empirical Risk and Risk Monotonicity

arXiv: https://arxiv.org/abs/1907.05476 · HTML: https://mlsanity.com/paper/1907.05476/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jul 2019.

## Built on by (most cited first, 11 of 11)

- [The Shape of Learning Curves: a Review](https://mlsanity.com/paper/2103.10948/) (2021)
- [Data augmentation in Bayesian neural networks and the cold posterior effect](https://mlsanity.com/paper/2106.05586/) (2021)
- [Learning Curves for Decision Making in Supervised Machine Learning: A Survey](https://mlsanity.com/paper/2201.12150/) (2022)
- [The no-free-lunch theorems of supervised learning](https://mlsanity.com/paper/2202.04513/) (2022)
- [Learning curves for Gaussian process regression with power-law priors and targets](https://mlsanity.com/paper/2110.12231/) (2021)
- [On Learning-Curve Monotonicity for Maximum Likelihood Estimators](https://mlsanity.com/paper/2512.10220/) (2025)
- [A Survey of Learning Curves with Bad Behavior: or How More Data Need Not Lead to Better Performance](https://mlsanity.com/paper/2211.14061/) (2022)
- [Monotonic Learning in the PAC Framework: A New Perspective](https://mlsanity.com/paper/2501.05493/) (2025)
- [LCDB 1.1: A Database Illustrating Learning Curves Are More Ill-Behaved Than Previously Thought](https://mlsanity.com/paper/2505.15657/) (2025)
- [Origins and mitigation of double descent in reduced order modeling](https://mlsanity.com/paper/2607.26414/) (2026)
- [The Impossible Trinity of Time-Series Validation: A Conservation Law among Training Sufficiency, Test Coverage, and Temporal Causality](https://mlsanity.com/paper/2609.29530/) (2026)

## Abstract (arXiv)

Plotting a learner's average performance against the number of training samples results in a learning curve. Studying such curves on one or more data sets is a way to get to a better understanding of the generalization properties of this learner. The behavior of learning curves is, however, not very well understood and can display (for most researchers) quite unexpected behavior. Our work introduces the formal notion of risk monotonicity, which asks the risk to not deteriorate with increasing training set sizes in expectation over the training samples. We then present the surprising result that various standard learners, specifically those that minimize the empirical risk, can act nonmonotonically irrespective of the training sample size. We provide a theoretical underpinning for specific instantiations from classification, regression, and density estimation. Altogether, the proposed monotonicity notion opens up a whole new direction of research.