# Is the Performance of My Deep Network Too Good to Be True? A Direct Approach to Estimating the Bayes Error in Binary Classification

arXiv: https://arxiv.org/abs/2202.00395 · HTML: https://mlsanity.com/paper/2202.00395/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 64 references that are in the map.
- First posted on arXiv 1 Feb 2022.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Scaling Laws for Autoregressive Generative Modeling](https://mlsanity.com/paper/2010.14701/) (2020)
- [Dynabench: Rethinking Benchmarking in NLP](https://mlsanity.com/paper/2104.14337/) (2021)
- [Are we done with ImageNet?](https://mlsanity.com/paper/2006.07159/) (2020)
- [Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations](https://mlsanity.com/paper/2110.12088/) (2021)
- [Human uncertainty makes classification more robust](https://mlsanity.com/paper/1908.07086/) (2019)
- [Do CIFAR-10 Classifiers Generalize to CIFAR-10?](https://mlsanity.com/paper/1806.00451/) (2018)
- [Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets](https://mlsanity.com/paper/2008.02637/) (2020)
- [Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labels](https://mlsanity.com/paper/2101.05022/) (2021)
- [Do We Train on Test Data? Purging CIFAR of Near-Duplicates](https://mlsanity.com/paper/1902.00423/) (2019)
- [Cold Case: The Lost MNIST Digits](https://mlsanity.com/paper/1905.10498/) (2019)
- [On the Minimal Supervision for Training Any Binary Classifier from Only Unlabeled Data](https://mlsanity.com/paper/1808.10585/) (2018)
- [Model Similarity Mitigates Test Set Overuse](https://mlsanity.com/paper/1905.12580/) (2019)
- [Classification from Pairwise Similarity and Unlabeled Data](https://mlsanity.com/paper/1802.04381/) (2018)
- [Binary Classification from Positive-Confidence Data](https://mlsanity.com/paper/1710.07138/) (2017)
- [Evaluating State-of-the-Art Classification Models Against Bayes Optimality](https://mlsanity.com/paper/2106.03357/) (2021)
- [Detecting Overfitting via Adversarial Examples](https://mlsanity.com/paper/1903.02380/) (2019)
- [Empirically Estimable Classification Bounds Based on a New Divergence Measure](https://mlsanity.com/paper/1412.6534/) (2014)
- [Learning to Benchmark: Determining Best Achievable Misclassification Error from Training Data](https://mlsanity.com/paper/1909.07192/) (2019)
- [Learning to Bound the Multi-class Bayes Error](https://mlsanity.com/paper/1811.06419/) (2018)

## Built on by (most cited first, 26 of 26)

- [Binary Classification with Confidence Difference](https://mlsanity.com/paper/2310.05632/) (2023)
- [Beyond Probability Partitions: Calibrating Neural Networks with Semantic Aware Grouping](https://mlsanity.com/paper/2306.04985/) (2023)
- [A General Framework for Learning from Weak Supervision](https://mlsanity.com/paper/2402.01922/) (2024)
- [Certified Robust Accuracy of Neural Networks Are Bounded due to Bayes Errors](https://mlsanity.com/paper/2405.11547/) (2024)
- [CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfitting](https://mlsanity.com/paper/2505.18102/) (2025)
- [Practical estimation of the optimal classification error with soft labels and calibration](https://mlsanity.com/paper/2505.20761/) (2025)
- [Unified Risk Analysis for Weakly Supervised Learning](https://mlsanity.com/paper/2309.08216/) (2023)
- [Data-Driven Estimation of the False Positive Rate of the Bayes Binary Classifier via Soft Labels](https://mlsanity.com/paper/2401.15500/) (2024)
- [Soft ascent-descent as a stable and flexible alternative to flooding](https://mlsanity.com/paper/2310.10006/) (2023)
- [How Does Bayes Error Limit Probabilistic Robust Accuracy](https://mlsanity.com/paper/2405.14923/) (2024)
- [Probabilistic Variational Contrastive Learning](https://mlsanity.com/paper/2506.10159/) (2025)
- [L2CU: Learning to Complement Unseen Users](https://mlsanity.com/paper/2601.06119/) (2026)
- [BOLT: Universal Classifier Training via a Bayes-Error Upper Bound](https://mlsanity.com/paper/2501.07754/) (2025)
- [Bayes Error Rate Estimation in Difficult Situations](https://mlsanity.com/paper/2506.03159/) (2025)
- [Generalizing Supervised Contrastive learning: A Projection Perspective](https://mlsanity.com/paper/2506.09810/) (2025)
- [Bounding Neyman-Pearson Region with f-Divergences](https://mlsanity.com/paper/2505.08899/) (2025)
- [Revisiting GAN with Bayes-Optimal Discrimination](https://mlsanity.com/paper/2510.25609/) (2025)
- [Robust and Explainable Divide-and-Conquer Learning for Intrusion Detection](https://mlsanity.com/paper/2605.02015/) (2026)
- [The Calibration Channel Determines the Bayes-Error Proxy: An Exact Law for Temperature-Induced Distortion](https://mlsanity.com/paper/2607.18162/) (2026)
- [Recent advances in weakly supervised learning: New supervision paradigms, assumption relaxations, and practical solutions](https://mlsanity.com/paper/2608.06896/) (2026)
- [Bias-Corrected Ceilings of Emotion Predictability from Human Label Variation Based on Instance-Level Fano Bounds](https://mlsanity.com/paper/2608.15619/) (2026)
- [Counterfactual Evaluation of Temporal Observation Protocols](https://mlsanity.com/paper/2608.22221/) (2026)
- [Bayes-Optimal BER and AUC: Estimation and Evaluation of Estimators](https://mlsanity.com/paper/2609.02304/) (2026)
- [AUC Maximization from Biased Positive-unlabeled Data with Confidence](https://mlsanity.com/paper/2609.10928/) (2026)
- [Decomposing A Neural Learner](https://mlsanity.com/paper/2610.14388/) (2026)
- [ProbabilityBench: Model Evaluation and Calibration with Exact Ground-Truth Uncertainty](https://mlsanity.com/paper/2610.17016/) (2026)

## Abstract (arXiv)

There is a fundamental limitation in the prediction performance that a machine learning model can achieve due to the inevitable uncertainty of the prediction target. In classification problems, this can be characterized by the Bayes error, which is the best achievable error with any classifier. The Bayes error can be used as a criterion to evaluate classifiers with state-of-the-art performance and can be used to detect test set overfitting. We propose a simple and direct Bayes error estimator, where we just take the mean of the labels that show uncertainty of the class assignments. Our flexible approach enables us to perform Bayes error estimation even for weakly supervised data. In contrast to others, our method is model-free and even instance-free. Moreover, it has no hyperparameters and gives a more accurate estimate of the Bayes error than several baselines empirically. Experiments using our method suggest that recently proposed deep networks such as the Vision Transformer may have reached, or is about to reach, the Bayes error for benchmark datasets. Finally, we discuss how we can study the inherent difficulty of the acceptance/rejection decision for scientific articles, by estimating the Bayes error of the ICLR papers from 2017 to 2023.