# Stationary Activations for Uncertainty Calibration in Deep Learning

arXiv: https://arxiv.org/abs/2010.09494 · HTML: https://mlsanity.com/paper/2010.09494/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Oct 2020.

## Built on by (most cited first, 14 of 14)

- [A Stitch in Time Saves Nine: A Train-Time Regularizing Loss for Improved Neural Network Calibration](https://mlsanity.com/paper/2203.13834/) (2022)
- [Deep Neural Networks as Point Estimates for Deep Gaussian Processes](https://mlsanity.com/paper/2105.04504/) (2021)
- [Towards Improving Calibration in Object Detection Under Domain Shift](https://mlsanity.com/paper/2209.07601/) (2022)
- [Periodic Activation Functions Induce Stationarity](https://mlsanity.com/paper/2110.13572/) (2021)
- [Fixing Overconfidence in Dynamic Neural Networks](https://mlsanity.com/paper/2302.06359/) (2023)
- [Detection of out-of-distribution samples using binary neuron activation patterns](https://mlsanity.com/paper/2212.14268/) (2022)
- [Squared Neural Families: A New Class of Tractable Density Models](https://mlsanity.com/paper/2305.13552/) (2023)
- [Beyond Unimodal: Generalising Neural Processes for Multimodal Uncertainty Estimation](https://mlsanity.com/paper/2304.01518/) (2023)
- [Greedy Bayesian Posterior Approximation with Deep Ensembles](https://mlsanity.com/paper/2105.14275/) (2021)
- [Revisiting the Equivalence of Bayesian Neural Networks and Gaussian Processes: On the Importance of Learning Activations](https://mlsanity.com/paper/2410.15777/) (2024)
- [Sparse Function-space Representation of Neural Networks](https://mlsanity.com/paper/2309.02195/) (2023)
- [Multiclass Alignment of Confidence and Certainty for Network Calibration](https://mlsanity.com/paper/2309.02636/) (2023)
- [Few-shot time series segmentation using prototype-defined infinite hidden Markov models](https://mlsanity.com/paper/2102.03885/) (2021)
- [Hi-fi functional priors by learning activations](https://mlsanity.com/paper/2508.08880/) (2025)

## Abstract (arXiv)

We introduce a new family of non-linear neural network activation functions that mimic the properties induced by the widely-used Mat'ern family of kernels in Gaussian process (GP) models. This class spans a range of locally stationary models of various degrees of mean-square differentiability. We show an explicit link to the corresponding GP models in the case that the network consists of one infinitely wide hidden layer. In the limit of infinite smoothness the Mat'ern family results in the RBF kernel, and in this case we recover RBF activations. Mat'ern activation functions result in similar appealing properties to their counterparts in GP models, and we demonstrate that the local stationarity property together with limited mean-square differentiability shows both good performance and uncertainty calibration in Bayesian deep learning tasks. In particular, local stationarity helps calibrate out-of-distribution (OOD) uncertainty. We demonstrate these properties on classification and regression benchmarks and a radar emitter classification task.