# Hi-fi functional priors by learning activations

arXiv: https://arxiv.org/abs/2508.08880 · HTML: https://mlsanity.com/paper/2508.08880/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 26 references that are in the map.
- First posted on arXiv 12 Aug 2025.

## Builds on (most cited first, 17 of 17)

- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [Stochastic Gradient Hamiltonian Monte Carlo](https://mlsanity.com/paper/1402.4102/) (2014)
- [Gaussian Process Behaviour in Wide Deep Neural Networks](https://mlsanity.com/paper/1804.11271/) (2018)
- [Functional Variational Bayesian Neural Networks](https://mlsanity.com/paper/1903.05779/) (2019)
- [Priors in Bayesian Deep Learning: A Review](https://mlsanity.com/paper/2105.06868/) (2021)
- [Pad'e Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networks](https://mlsanity.com/paper/1907.06732/) (2019)
- [All You Need is a Good Functional Prior for Bayesian Deep Learning](https://mlsanity.com/paper/2011.12829/) (2020)
- [Exact posterior distributions of wide Bayesian neural networks](https://mlsanity.com/paper/2006.10541/) (2020)
- [Expressive Priors in Bayesian Neural Networks: Kernel Combinations and Periodic Functions](https://mlsanity.com/paper/1905.06076/) (2019)
- [Periodic Activation Functions Induce Stationarity](https://mlsanity.com/paper/2110.13572/) (2021)
- [The Ridgelet Prior: A Covariance Function Approach to Prior Specification for Bayesian Neural Networks](https://mlsanity.com/paper/2010.08488/) (2020)
- [Stationary Activations for Uncertainty Calibration in Deep Learning](https://mlsanity.com/paper/2010.09494/) (2020)
- [Hierarchical Gaussian Process Priors for Bayesian Neural Network Weights](https://mlsanity.com/paper/2002.04033/) (2020)
- [Should We Learn Most Likely Functions or Parameters?](https://mlsanity.com/paper/2311.15990/) (2023)
- [Richer priors for infinitely wide multi-layer perceptrons](https://mlsanity.com/paper/1911.12927/) (2019)
- [Wide Bayesian neural networks have a simple weight posterior: theory and accelerated sampling](https://mlsanity.com/paper/2206.07673/) (2022)

## Abstract (arXiv)

Function-space priors in Bayesian Neural Networks (BNNs) provide a more intuitive approach to embedding beliefs directly into the model's output, thereby enhancing regularization, uncertainty quantification, and risk-aware decision-making. However, imposing function-space priors on BNNs is challenging. We address this task through optimization techniques that explore how trainable activations can accommodate higher-complexity priors and match intricate target function distributions. We investigate flexible activation models, including Pade functions and piecewise linear functions, and discuss the learning challenges related to identifiability, loss construction, and symmetries. Our empirical findings indicate that even BNNs with a single wide hidden layer when equipped with flexible trainable activation, can effectively achieve desired function-space priors.