# Single Layer Predictive Normalized Maximum Likelihood for Out-of-Distribution Detection

arXiv: https://arxiv.org/abs/2110.09246 · HTML: https://mlsanity.com/paper/2110.09246/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 47 references that are in the map.
- First posted on arXiv 18 Oct 2021.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Spectrally-normalized margin bounds for neural networks](https://mlsanity.com/paper/1706.08498/) (2017)
- [Predictive Uncertainty Estimation via Prior Networks](https://mlsanity.com/paper/1802.10501/) (2018)
- [Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1906.12340/) (2019)
- [Fantastic Generalization Measures and Where to Find Them](https://mlsanity.com/paper/1912.02178/) (2019)
- [A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks](https://mlsanity.com/paper/1707.09564/) (2017)
- [TurkerGaze: Crowdsourcing Saliency with Webcam based Eye Tracking](https://mlsanity.com/paper/1504.06755/) (2015)
- [Out-of-Distribution Detection Using an Ensemble of Self Supervised Leave-out Classifiers](https://mlsanity.com/paper/1809.03576/) (2018)
- [Recovery Guarantees for One-hidden-layer Neural Networks](https://mlsanity.com/paper/1706.03175/) (2017)
- [Towards Maximizing the Representation Gap between In-Domain & Out-of-Distribution Examples](https://mlsanity.com/paper/2010.10474/) (2020)
- [Offline Model-Based Optimization via Normalized Maximum Likelihood Estimation](https://mlsanity.com/paper/2102.07970/) (2021)
- [Safety Concerns and Mitigation Approaches Regarding the Use of Deep Learning in Safety-Critical Perception Tasks](https://mlsanity.com/paper/2001.08001/) (2020)
- [Deep pNML: Predictive Normalized Maximum Likelihood for Deep Neural Networks](https://mlsanity.com/paper/1904.12286/) (2019)
- [A New Look at an Old Problem: A Universal Learning Approach to Linear Regression](https://mlsanity.com/paper/1905.04708/) (2019)
- [Uncertainty aware and explainable diagnosis of retinal disease](https://mlsanity.com/paper/2101.12041/) (2021)
- [Balancing Specialization, Generalization, and Compression for Detection and Tracking](https://mlsanity.com/paper/1909.11348/) (2019)
- [Amortized Conditional Normalized Maximum Likelihood: Reliable Out of Distribution Uncertainty Estimation](https://mlsanity.com/paper/2011.02696/) (2020)
- [Learning, compression, and leakage: Minimising classification error via meta-universal compression principles](https://mlsanity.com/paper/2010.07382/) (2020)
- [Learning Rotation Invariant Features for Cryogenic Electron Microscopy Image Reconstruction](https://mlsanity.com/paper/2101.03549/) (2021)

## Built on by (most cited first, 23 of 23)

- [Generalized Out-of-Distribution Detection: A Survey](https://mlsanity.com/paper/2110.11334/) (2021)
- [RankFeat: Rank-1 Feature Removal for Out-of-distribution Detection](https://mlsanity.com/paper/2209.08590/) (2022)
- [AIROGS: Artificial Intelligence for RObust Glaucoma Screening Challenge](https://mlsanity.com/paper/2302.01738/) (2023)
- [FiLM-Ensemble: Probabilistic Deep Learning via Feature-wise Linear Modulation](https://mlsanity.com/paper/2206.00050/) (2022)
- [Few-shot Learning with Noisy Labels](https://mlsanity.com/paper/2204.05494/) (2022)
- [Is Retain Set All You Need in Machine Unlearning? Restoring Performance of Unlearned Models with Out-Of-Distribution Images](https://mlsanity.com/paper/2404.12922/) (2024)
- [Adaptive Hierarchical Graph Cut for Multi-granularity Out-of-distribution Detection](https://mlsanity.com/paper/2412.15668/) (2024)
- [DeepLens: Interactive Out-of-distribution Data Detection in NLP Models](https://mlsanity.com/paper/2303.01577/) (2023)
- [SUDO: a framework for evaluating clinical artificial intelligence systems without ground-truth annotations](https://mlsanity.com/paper/2403.17011/) (2024)
- [Hierarchical Visual Categories Modeling: A Joint Representation Learning and Density Estimation Framework for Out-of-Distribution Detection](https://mlsanity.com/paper/2408.15580/) (2024)
- [Collaborative Image Understanding](https://mlsanity.com/paper/2210.11907/) (2022)
- [HyperMix: Out-of-Distribution Detection and Classification in Few-Shot Settings](https://mlsanity.com/paper/2312.15086/) (2023)
- [TagOOD: A Novel Approach to Out-of-Distribution Detection via Vision-Language Representations and Class Center Learning](https://mlsanity.com/paper/2408.15566/) (2024)
- [Reflexive Guidance: Improving OoDD in Vision-Language Models via Self-Guided Image-Adaptive Concept Generation](https://mlsanity.com/paper/2410.14975/) (2024)
- [QuantProb: Generalizing Probabilities along with Predictions for a Pre-trained Classifier](https://mlsanity.com/paper/2304.12766/) (2023)
- [RankFeat&RankWeight: Rank-1 Feature/Weight Removal for Out-of-distribution Detection](https://mlsanity.com/paper/2311.13959/) (2023)
- [Hypercone Assisted Contour Generation for Out-of-Distribution Detection](https://mlsanity.com/paper/2501.10209/) (2025)
- [Beyond Ridge Regression for Distribution-Free Data](https://mlsanity.com/paper/2206.08757/) (2022)
- [Two-stage Modeling for Prediction with Confidence](https://mlsanity.com/paper/2209.08848/) (2022)
- [Learning to Generate Parameters of ConvNets for Unseen Image Data](https://mlsanity.com/paper/2310.11862/) (2023)
- [Distilling the Unknown to Unveil Certainty](https://mlsanity.com/paper/2311.07975/) (2023)
- [A Systematic Analysis of Out-of-Distribution Detection Under Representation and Training Paradigm Shifts](https://mlsanity.com/paper/2511.11934/) (2025)
- [Training-Free Out-of-Distribution Detection for Pathology Whole-Slide Images](https://mlsanity.com/paper/2608.01407/) (2026)

## Abstract (arXiv)

Detecting out-of-distribution (OOD) samples is vital for developing machine learning based models for critical safety systems. Common approaches for OOD detection assume access to some OOD samples during training which may not be available in a real-life scenario. Instead, we utilize the em predictive normalized maximum likelihood (pNML) learner, in which no assumptions are made on the tested input. We derive an explicit expression of the pNML and its generalization error, denoted as the em regret, for a single layer neural network (NN). We show that this learner generalizes well when (i) the test vector resides in a subspace spanned by the eigenvectors associated with the large eigenvalues of the empirical correlation matrix of the training data, or (ii) the test sample is far from the decision boundary. Furthermore, we describe how to efficiently apply the derived pNML regret to any pretrained deep NN, by employing the explicit pNML for the last layer, followed by the softmax function. Applying the derived regret to deep NN requires neither additional tunable parameters nor extra data. We extensively evaluate our approach on 74 OOD detection benchmarks using DenseNet-100, ResNet-34, and WideResNet-40 models trained with CIFAR-100, CIFAR-10, SVHN, and ImageNet-30 showing a significant improvement of up to 15.6% over recent leading methods.