# Statistical Hypothesis Testing Based on Machine Learning: Large Deviations Analysis

arXiv: https://arxiv.org/abs/2207.10939 · HTML: https://mlsanity.com/paper/2207.10939/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 110 references that are in the map.
- First posted on arXiv 22 Jul 2022.

## Builds on (most cited first, 28 of 28)

- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Least Squares Generative Adversarial Networks](https://mlsanity.com/paper/1611.04076/) (2016)
- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [Object Detection with Deep Learning: A Review](https://mlsanity.com/paper/1807.05511/) (2018)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [A Survey of Deep Learning Techniques for Autonomous Driving](https://mlsanity.com/paper/1910.07738/) (2019)
- [Tracking without bells and whistles](https://mlsanity.com/paper/1903.05625/) (2019)
- [Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes](https://mlsanity.com/paper/1810.05148/) (2018)
- [Understanding Deep Convolutional Networks](https://mlsanity.com/paper/1601.04920/) (2016)
- [Deep Learning COVID-19 Features on CXR using Limited Training Data Sets](https://mlsanity.com/paper/2004.05758/) (2020)
- [Learning to Detect](https://mlsanity.com/paper/1805.07631/) (2018)
- [Deep Scattering Spectrum](https://mlsanity.com/paper/1304.6763/) (2013)
- [Model-Based Deep Learning: On the Intersection of Deep Learning and Optimization](https://mlsanity.com/paper/2205.02640/) (2022)
- [Deep Learning Methods for Vessel Trajectory Prediction based on Recurrent Neural Networks](https://mlsanity.com/paper/2101.02486/) (2021)
- [Understanding Autoencoders with Information Theoretic Concepts](https://mlsanity.com/paper/1804.00057/) (2018)
- [Lessons from AlphaZero for Optimal, Model Predictive, and Adaptive Control](https://mlsanity.com/paper/2108.10315/) (2021)
- [Training Neural Networks for Likelihood/Density Ratio Estimation](https://mlsanity.com/paper/1911.00405/) (2019)
- [Active Sensing for Communications by Learning](https://mlsanity.com/paper/2112.04075/) (2021)
- [Space-based Global Maritime Surveillance. Part I: Satellite Technologies](https://mlsanity.com/paper/2011.11304/) (2020)
- [Model-Based Machine Learning for Communications](https://mlsanity.com/paper/2101.04726/) (2021)
- [Space-based Global Maritime Surveillance. Part II: Artificial Intelligence and Data Fusion Techniques](https://mlsanity.com/paper/2011.11338/) (2020)
- [Distributed Detection over Adaptive Networks: Refined Asymptotics and the Role of Connectivity](https://mlsanity.com/paper/1601.07011/) (2016)
- [Classification-Aided Multitarget Tracking Using the Sum-Product Algorithm](https://mlsanity.com/paper/2008.01667/) (2020)
- [Recurrent Encoder-Decoder Networks for Vessel Trajectory Prediction with Uncertainty Estimation](https://mlsanity.com/paper/2205.05404/) (2022)
- [Quickest Detection and Forecast of Pandemic Outbreaks: Analysis of COVID-19 Waves](https://mlsanity.com/paper/2101.04620/) (2021)

## Built on by (most cited first, 6 of 6)

- [CFARnet: deep learning for target detection with constant false alarm rate](https://mlsanity.com/paper/2208.02474/) (2022)
- [Accurate Passive Radar via an Uncertainty-Aware Fusion of Wi-Fi Sensing Data](https://mlsanity.com/paper/2407.04733/) (2024)
- [Non-Asymptotic Performance of Social Machine Learning Under Limited Data](https://mlsanity.com/paper/2306.09397/) (2023)
- [Tight Long-Term Tail Decay of (Clipped) SGD in Non-Convex Optimization](https://mlsanity.com/paper/2602.05657/) (2026)
- [Large Deviations for Classification Performance Analysis of Machine Learning Systems](https://mlsanity.com/paper/2301.07104/) (2023)
- [Extended target tracking utilizing machine-learning software -- with applications to animal classification](https://mlsanity.com/paper/2310.08316/) (2023)

## Abstract (arXiv)

We study the performance -- and specifically the rate at which the error probability converges to zero -- of Machine Learning (ML) classification techniques. Leveraging the theory of large deviations, we provide the mathematical conditions for a ML classifier to exhibit error probabilities that vanish exponentially, say ~ expleft(-n,I + o(n) right), where n is the number of informative observations available for testing (or another relevant parameter, such as the size of the target in an image) and I is the error rate. Such conditions depend on the Fenchel-Legendre transform of the cumulant-generating function of the Data-Driven Decision Function (D3F, i.e., what is thresholded before the final binary decision is made) learned in the training phase. As such, the D3F and, consequently, the related error rate I, depend on the given training set, which is assumed of finite size. Interestingly, these conditions can be verified and tested numerically exploiting the available dataset, or a synthetic dataset, generated according to the available information on the underlying statistical model. In other words, the classification error probability convergence to zero and its rate can be computed on a portion of the dataset available for training. Coherently with the large deviations theory, we can also establish the convergence, for n large enough, of the normalized D3F statistic to a Gaussian distribution. This property is exploited to set a desired asymptotic false alarm probability, which empirically turns out to be accurate even for quite realistic values of n. Furthermore, approximate error probability curves ~ zetan expleft(-n,I right) are provided, thanks to the refined asymptotic derivation (often referred to as exact asymptotics), where zetan represents the most representative sub-exponential terms of the error probabilities.