# Learning to Benchmark: Determining Best Achievable Misclassification Error from Training Data

arXiv: https://arxiv.org/abs/1909.07192 · HTML: https://mlsanity.com/paper/1909.07192/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Sep 2019.

## Built on by (most cited first, 11 of 11)

- [Is the Performance of My Deep Network Too Good to Be True? A Direct Approach to Estimating the Bayes Error in Binary Classification](https://mlsanity.com/paper/2202.00395/) (2022)
- [Evaluating State-of-the-Art Classification Models Against Bayes Optimality](https://mlsanity.com/paper/2106.03357/) (2021)
- [Certified Robust Accuracy of Neural Networks Are Bounded due to Bayes Errors](https://mlsanity.com/paper/2405.11547/) (2024)
- [Practical estimation of the optimal classification error with soft labels and calibration](https://mlsanity.com/paper/2505.20761/) (2025)
- [How Does Bayes Error Limit Probabilistic Robust Accuracy](https://mlsanity.com/paper/2405.14923/) (2024)
- [BOLT: Universal Classifier Training via a Bayes-Error Upper Bound](https://mlsanity.com/paper/2501.07754/) (2025)
- [Bayes Error Rate Estimation in Difficult Situations](https://mlsanity.com/paper/2506.03159/) (2025)
- [Bounding Neyman-Pearson Region with f-Divergences](https://mlsanity.com/paper/2505.08899/) (2025)
- [Revisiting GAN with Bayes-Optimal Discrimination](https://mlsanity.com/paper/2510.25609/) (2025)
- [Bayes-Optimal BER and AUC: Estimation and Evaluation of Estimators](https://mlsanity.com/paper/2609.02304/) (2026)
- [ProbabilityBench: Model Evaluation and Calibration with Exact Ground-Truth Uncertainty](https://mlsanity.com/paper/2610.17016/) (2026)

## Abstract (arXiv)

We address the problem of learning to benchmark the best achievable classifier performance. In this problem the objective is to establish statistically consistent estimates of the Bayes misclassification error rate without having to learn a Bayes-optimal classifier. Our learning to benchmark framework improves on previous work on learning bounds on Bayes misclassification rate since it learns the it exact Bayes error rate instead of a bound on error rate. We propose a benchmark learner based on an ensemble of ε-ball estimators and Chebyshev approximation. Under a smoothness assumption on the class densities we show that our estimator achieves an optimal (parametric) mean squared error (MSE) rate of O(N-1), where N is the number of samples. Experiments on both simulated and real datasets establish that our proposed benchmark learning algorithm produces estimates of the Bayes error that are more accurate than previous approaches for learning bounds on Bayes error probability.