# Deep pNML: Predictive Normalized Maximum Likelihood for Deep Neural Networks

arXiv: https://arxiv.org/abs/1904.12286 · HTML: https://mlsanity.com/paper/1904.12286/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Apr 2019.

## Built on by (most cited first, 12 of 12)

- [Offline Model-Based Optimization via Normalized Maximum Likelihood Estimation](https://mlsanity.com/paper/2102.07970/) (2021)
- [MURAL: Meta-Learning Uncertainty-Aware Rewards for Outcome-Driven Reinforcement Learning](https://mlsanity.com/paper/2107.07184/) (2021)
- [Single Layer Predictive Normalized Maximum Likelihood for Out-of-Distribution Detection](https://mlsanity.com/paper/2110.09246/) (2021)
- [Introspective Learning : A Two-Stage Approach for Inference in Neural Networks](https://mlsanity.com/paper/2209.08425/) (2022)
- [Outcome-directed Reinforcement Learning by Uncertainty & Temporal Distance-Aware Curriculum Goal Generation](https://mlsanity.com/paper/2301.11741/) (2023)
- [Distribution Free Uncertainty for the Minimum Norm Solution of Over-parameterized Linear Regression](https://mlsanity.com/paper/2102.07181/) (2021)
- [Utilizing Adversarial Targeted Attacks to Boost Adversarial Robustness](https://mlsanity.com/paper/2109.01945/) (2021)
- [Collaborative Image Understanding](https://mlsanity.com/paper/2210.11907/) (2022)
- [Measuring Stochastic Data Complexity with Boltzmann Influence Functions](https://mlsanity.com/paper/2406.02745/) (2024)
- [Beyond Ridge Regression for Distribution-Free Data](https://mlsanity.com/paper/2206.08757/) (2022)
- [Universal Batch Learning Under The Misspecification Setting](https://mlsanity.com/paper/2405.07252/) (2024)
- [Quantifying the Prediction Uncertainty of Machine Learning Models for Individual Data](https://mlsanity.com/paper/2412.07520/) (2024)

## Abstract (arXiv)

The Predictive Normalized Maximum Likelihood (pNML) scheme has been recently suggested for universal learning in the individual setting, where both the training and test samples are individual data. The goal of universal learning is to compete with a ``genie'' or reference learner that knows the data values, but is restricted to use a learner from a given model class. The pNML minimizes the associated regret for any possible value of the unknown label. Furthermore, its min-max regret can serve as a pointwise measure of learnability for the specific training and data sample. In this work we examine the pNML and its associated learnability measure for the Deep Neural Network (DNN) model class. As shown, the pNML outperforms the commonly used Empirical Risk Minimization (ERM) approach and provides robustness against adversarial attacks. Together with its learnability measure it can detect out of distribution test examples, be tolerant to noisy labels and serve as a confidence measure for the ERM. Finally, we extend the pNML to a ``twice universal'' solution, that provides universality for model class selection and generates a learner competing with the best one from all model classes.