# Function Space Diversity for Uncertainty Prediction via Repulsive Last-Layer Ensembles

arXiv: https://arxiv.org/abs/2412.15758 · HTML: https://mlsanity.com/paper/2412.15758/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 69 references that are in the map.
- First posted on arXiv 20 Dec 2024.

## Builds on (most cited first, 30 of 42)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [Deep Learning for Classical Japanese Literature](https://mlsanity.com/paper/1812.01718/) (2018)
- [Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness](https://mlsanity.com/paper/2006.10108/) (2020)
- [Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks](https://mlsanity.com/paper/2002.10118/) (2020)
- [Knowledge Distillation by On-the-Fly Native Ensemble](https://mlsanity.com/paper/1806.04606/) (2018)
- [Contrastive Training for Improved Out-of-Distribution Detection](https://mlsanity.com/paper/2007.05566/) (2020)
- [Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Counts](https://mlsanity.com/paper/2006.09239/) (2020)
- [Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks](https://mlsanity.com/paper/1511.06314/) (2015)
- [Functional Variational Bayesian Neural Networks](https://mlsanity.com/paper/1903.05779/) (2019)
- [Quantifying Aleatoric and Epistemic Uncertainty in Machine Learning: Are Conditional Entropy and Mutual Information Appropriate Measures?](https://mlsanity.com/paper/2209.03302/) (2022)
- [Repulsive Deep Ensembles are Bayesian](https://mlsanity.com/paper/2106.11642/) (2021)
- [On Feature Collapse and Deep Kernel Learning for Single Forward Pass Uncertainty](https://mlsanity.com/paper/2102.11409/) (2021)
- [Collaborative Learning for Deep Neural Networks](https://mlsanity.com/paper/1805.11761/) (2018)
- [The Case for Bayesian Deep Learning](https://mlsanity.com/paper/2001.10995/) (2020)
- [Variational Bayesian Last Layers](https://mlsanity.com/paper/2404.11599/) (2024)
- [Do Bayesian Neural Networks Need To Be Fully Stochastic?](https://mlsanity.com/paper/2211.06291/) (2022)
- [On the Practicality of Deterministic Epistemic Uncertainty](https://mlsanity.com/paper/2107.00649/) (2021)
- [A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awareness](https://mlsanity.com/paper/2205.00403/) (2022)
- [A Unified Particle-Optimization Framework for Scalable Bayesian Sampling](https://mlsanity.com/paper/1805.11659/) (2018)
- [Function Space Particle Optimization for Bayesian Neural Networks](https://mlsanity.com/paper/1902.09754/) (2019)
- [Tractable Function-Space Variational Inference in Bayesian Neural Networks](https://mlsanity.com/paper/2312.17199/) (2023)

## Built on by (most cited first, 8 of 8)

- [Uncertainty Reasoning with Photonic Bayesian Machines](https://mlsanity.com/paper/2512.02217/) (2025)
- [Uncertainty Estimation using Variance-Gated Distributions](https://mlsanity.com/paper/2509.08846/) (2025)
- [Variance-Gated Ensembles: An Epistemic-Aware Framework for Uncertainty Estimation](https://mlsanity.com/paper/2602.08142/) (2026)
- [Latent Uncertainty Representations for Video-based Driver Action and Intention Recognition](https://mlsanity.com/paper/2510.05006/) (2025)
- [Accelerated Execution of Bayesian Neural Networks using a Single Probabilistic Forward Pass and Code Generation](https://mlsanity.com/paper/2511.23440/) (2025)
- [How (Mis)calibrated is your Federated CLIP and what to do about it?](https://mlsanity.com/paper/2512.04305/) (2025)
- [Controllable Diversity in Normalization-Based Implicit Ensembles via Softmax-Temperature Modulation](https://mlsanity.com/paper/2607.23860/) (2026)
- [Uncertainty quantification for trustworthy deep learning: Methods and measures](https://mlsanity.com/paper/2607.28248/) (2026)

## Abstract (arXiv)

Bayesian inference in function space has gained attention due to its robustness against overparameterization in neural networks. However, approximating the infinite-dimensional function space introduces several challenges. In this work, we discuss function space inference via particle optimization and present practical modifications that improve uncertainty estimation and, most importantly, make it applicable for large and pretrained networks. First, we demonstrate that the input samples, where particle predictions are enforced to be diverse, are detrimental to the model performance. While diversity on training data itself can lead to underfitting, the use of label-destroying data augmentation, or unlabeled out-of-distribution data can improve prediction diversity and uncertainty estimates. Furthermore, we take advantage of the function space formulation, which imposes no restrictions on network parameterization other than sufficient flexibility. Instead of using full deep ensembles to represent particles, we propose a single multi-headed network that introduces a minimal increase in parameters and computation. This allows seamless integration to pretrained networks, where this repulsive last-layer ensemble can be used for uncertainty aware fine-tuning at minimal additional cost. We achieve competitive results in disentangling aleatoric and epistemic uncertainty for active learning, detecting out-of-domain data, and providing calibrated uncertainty estimates under distribution shifts with minimal compute and memory.