# Function-Space Regularization in Neural Networks: A Probabilistic Perspective

arXiv: https://arxiv.org/abs/2312.17162 · HTML: https://mlsanity.com/paper/2312.17162/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 47 references that are in the map.
- First posted on arXiv 28 Dec 2023.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Bayesian Deep Learning and a Probabilistic Perspective of Generalization](https://mlsanity.com/paper/2002.08791/) (2020)
- [Deep Learning for Classical Japanese Literature](https://mlsanity.com/paper/1812.01718/) (2018)
- [Do CIFAR-10 Classifiers Generalize to CIFAR-10?](https://mlsanity.com/paper/1806.00451/) (2018)
- [Functional Variational Bayesian Neural Networks](https://mlsanity.com/paper/1903.05779/) (2019)
- [Functional Regularisation for Continual Learning with Gaussian Processes](https://mlsanity.com/paper/1901.11356/) (2019)
- [Plex: Towards Reliability using Pretrained Large Model Extensions](https://mlsanity.com/paper/2207.07411/) (2022)
- [Measuring and regularizing networks in function space](https://mlsanity.com/paper/1805.08289/) (2018)
- [Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning](https://mlsanity.com/paper/2106.04015/) (2021)
- [Benchmarking Bayesian Deep Learning on Diabetic Retinopathy Detection Tasks](https://mlsanity.com/paper/2211.12717/) (2022)
- [Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes](https://mlsanity.com/paper/2005.08140/) (2020)
- [Variational Implicit Processes](https://mlsanity.com/paper/1806.02390/) (2018)
- [Function Space Particle Optimization for Bayesian Neural Networks](https://mlsanity.com/paper/1902.09754/) (2019)
- [Understanding Variational Inference in Function-Space](https://mlsanity.com/paper/2011.09421/) (2020)
- [Does progress on ImageNet transfer to real-world datasets?](https://mlsanity.com/paper/2301.04644/) (2023)
- [Identifying Melanoma Images using EfficientNet Ensemble: Winning Solution to the SIIM-ISIC Melanoma Classification Challenge](https://mlsanity.com/paper/2010.05351/) (2020)
- [Selective Prediction via Training Dynamics](https://mlsanity.com/paper/2205.13532/) (2022)
- [iCassava 2019 Fine-Grained Visual Categorization Challenge](https://mlsanity.com/paper/1908.02900/) (2019)

## Built on by (most cited first, 20 of 20)

- [Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI](https://mlsanity.com/paper/2402.00809/) (2024)
- [Context-Guided Diffusion for Out-of-Distribution Molecular and Protein Design](https://mlsanity.com/paper/2407.11942/) (2024)
- [Models of Heavy-Tailed Mechanistic Universality](https://mlsanity.com/paper/2506.03470/) (2025)
- [Function Space Diversity for Uncertainty Prediction via Repulsive Last-Layer Ensembles](https://mlsanity.com/paper/2412.15758/) (2024)
- [A Gaussian Process View on Observation Noise and Initialization in Wide Neural Networks](https://mlsanity.com/paper/2502.01556/) (2025)
- [Deep Exploration with PAC-Bayes](https://mlsanity.com/paper/2402.03055/) (2024)
- [Spectral Convolutional Conditional Neural Processes](https://mlsanity.com/paper/2404.13182/) (2024)
- [Regularized KL-Divergence for Well-Defined Function-Space Variational Inference in Bayesian neural networks](https://mlsanity.com/paper/2406.04317/) (2024)
- [Attacking Bayes: On the Adversarial Robustness of Bayesian Neural Networks](https://mlsanity.com/paper/2404.19640/) (2024)
- [An Empirical Analysis of Calibration and Selective Prediction in Multimodal Clinical Condition Classification](https://mlsanity.com/paper/2603.02719/) (2026)
- [Sparse Deep Learning Models with the ℓ1 Regularization](https://mlsanity.com/paper/2408.02801/) (2024)
- [A Monte Carlo Framework for Calibrated Uncertainty Estimation in Sequence Prediction](https://mlsanity.com/paper/2410.23272/) (2024)
- [A neurosymbolic Approach with Epistemic Deep Learning for Hierarchical Image Classification](https://mlsanity.com/paper/2605.16383/) (2026)
- [Mind the GAP: Improving Robustness to Subpopulation Shifts with Group-Aware Priors](https://mlsanity.com/paper/2403.09869/) (2024)
- [Feed Two Birds with One Scone: Exploiting Function-Space Regularization for Both OOD Robustness and ID Fine-Tuning Performance](https://mlsanity.com/paper/2509.05328/) (2025)
- [Function-Space Empirical Bayes Regularisation with Large Vision-Language Model Priors](https://mlsanity.com/paper/2602.03119/) (2026)
- [Diffusion-Inspired Reconfiguration of Transformers for Uncertainty Calibration](https://mlsanity.com/paper/2602.08920/) (2026)
- [Function-Space Empirical Bayes Regularisation with Student's t Priors](https://mlsanity.com/paper/2602.22015/) (2026)
- [Data-Driven Priors for Uncertainty-Aware Risk Prediction of Clinical Deterioration using Multimodal Data](https://mlsanity.com/paper/2603.08459/) (2026)
- [Fora: From Weight-Space to Function-Space Protection in Capability-Preserving Fine-Tuning](https://mlsanity.com/paper/2606.31092/) (2026)

## Abstract (arXiv)

Parameter-space regularization in neural network optimization is a fundamental tool for improving generalization. However, standard parameter-space regularization methods make it challenging to encode explicit preferences about desired predictive functions into neural network training. In this work, we approach regularization in neural networks from a probabilistic perspective and show that by viewing parameter-space regularization as specifying an empirical prior distribution over the model parameters, we can derive a probabilistically well-motivated regularization technique that allows explicitly encoding information about desired predictive functions into neural network training. This method -- which we refer to as function-space empirical Bayes (FSEB) -- includes both parameter- and function-space regularization, is mathematically simple, easy to implement, and incurs only minimal computational overhead compared to standard regularization techniques. We evaluate the utility of this regularization technique empirically and demonstrate that the proposed method leads to near-perfect semantic shift detection, highly-calibrated predictive uncertainty estimates, successful task adaption from pre-trained models, and improved generalization under covariate shift.