# Evaluating State-of-the-Art Classification Models Against Bayes Optimality

arXiv: https://arxiv.org/abs/2106.03357 · HTML: https://mlsanity.com/paper/2106.03357/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 38 references that are in the map.
- First posted on arXiv 7 Jun 2021.

## Builds on (most cited first, 20 of 20)

- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [Deep Learning for Classical Japanese Literature](https://mlsanity.com/paper/1812.01718/) (2018)
- [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://mlsanity.com/paper/2103.14749/) (2021)
- [EMNIST: an extension of MNIST to handwritten letters](https://mlsanity.com/paper/1702.05373/) (2017)
- [Shake-Shake regularization](https://mlsanity.com/paper/1705.07485/) (2017)
- [SpinalNet: Deep Neural Network with Gradual Input](https://mlsanity.com/paper/2007.03347/) (2020)
- [Understanding the Limitations of Conditional Generative Models](https://mlsanity.com/paper/1906.01171/) (2019)
- [Fine-Tuning DARTS for Image Classification](https://mlsanity.com/paper/2006.09042/) (2020)
- [Integrals over Gaussians under Linear Domain Constraints](https://mlsanity.com/paper/1910.09328/) (2019)
- [Empirically Estimable Classification Bounds Based on a New Divergence Measure](https://mlsanity.com/paper/1412.6534/) (2014)
- [Generalized Bhattacharyya and Chernoff upper bounds on Bayes error using quasi-arithmetic means](https://mlsanity.com/paper/1401.4788/) (2014)
- [Pretrained AI Models: Performativity, Mobility, and Change](https://mlsanity.com/paper/1909.03290/) (2019)
- [Learning to Benchmark: Determining Best Achievable Misclassification Error from Training Data](https://mlsanity.com/paper/1909.07192/) (2019)
- [Multivariate f-Divergence Estimation With Confidence](https://mlsanity.com/paper/1411.2045/) (2014)
- [Meta learning of bounds on the Bayes classifier error](https://mlsanity.com/paper/1504.07116/) (2015)

## Built on by (most cited first, 16 of 16)

- [Is the Performance of My Deep Network Too Good to Be True? A Direct Approach to Estimating the Bayes Error in Binary Classification](https://mlsanity.com/paper/2202.00395/) (2022)
- [Certified Robust Accuracy of Neural Networks Are Bounded due to Bayes Errors](https://mlsanity.com/paper/2405.11547/) (2024)
- [CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfitting](https://mlsanity.com/paper/2505.18102/) (2025)
- [Practical estimation of the optimal classification error with soft labels and calibration](https://mlsanity.com/paper/2505.20761/) (2025)
- [How Does Bayes Error Limit Probabilistic Robust Accuracy](https://mlsanity.com/paper/2405.14923/) (2024)
- [On Optimal Steering to Achieve Exact Fairness](https://mlsanity.com/paper/2509.15759/) (2025)
- [Uncertainty Aware Neural Network from Similarity and Sensitivity](https://mlsanity.com/paper/2304.14925/) (2023)
- [A Fast, Robust Elliptical Slice Sampling Implementation for Linearly Truncated Multivariate Normal Distributions](https://mlsanity.com/paper/2407.10449/) (2024)
- [BOLT: Universal Classifier Training via a Bayes-Error Upper Bound](https://mlsanity.com/paper/2501.07754/) (2025)
- [Bayes Error Rate Estimation in Difficult Situations](https://mlsanity.com/paper/2506.03159/) (2025)
- [Bounding Neyman-Pearson Region with f-Divergences](https://mlsanity.com/paper/2505.08899/) (2025)
- [Towards Provably Unlearnable Examples via Bayes Error Optimisation](https://mlsanity.com/paper/2511.08191/) (2025)
- [Beyond Pooling: Matching for Robust Generalization under Data Heterogeneity](https://mlsanity.com/paper/2602.07154/) (2026)
- [The generalized underlap coefficient with an application in clustering](https://mlsanity.com/paper/2602.19473/) (2026)
- [Bayes-Optimal BER and AUC: Estimation and Evaluation of Estimators](https://mlsanity.com/paper/2609.02304/) (2026)
- [ProbabilityBench: Model Evaluation and Calibration with Exact Ground-Truth Uncertainty](https://mlsanity.com/paper/2610.17016/) (2026)

## Abstract (arXiv)

Evaluating the inherent difficulty of a given data-driven classification problem is important for establishing absolute benchmarks and evaluating progress in the field. To this end, a natural quantity to consider is the Bayes error, which measures the optimal classification error theoretically achievable for a given data distribution. While generally an intractable quantity, we show that we can compute the exact Bayes error of generative models learned using normalizing flows. Our technique relies on a fundamental result, which states that the Bayes error is invariant under invertible transformation. Therefore, we can compute the exact Bayes error of the learned flow models by computing it for Gaussian base distributions, which can be done efficiently using Holmes-Diaconis-Ross integration. Moreover, we show that by varying the temperature of the learned flow models, we can generate synthetic datasets that closely resemble standard benchmark datasets, but with almost any desired Bayes error. We use our approach to conduct a thorough investigation of state-of-the-art classification models, and find that in some -- but not all -- cases, these models are capable of obtaining accuracy very near optimal. Finally, we use our method to evaluate the intrinsic "hardness" of standard benchmark datasets, and classes within those datasets.