# Hamiltonian Monte Carlo on ReLU Neural Networks is Inefficient

arXiv: https://arxiv.org/abs/2410.22065 · HTML: https://mlsanity.com/paper/2410.22065/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 33 references that are in the map.
- First posted on arXiv 29 Oct 2024.

## Builds on (most cited first, 11 of 11)

- [Age Progression/Regression by Conditional Adversarial Autoencoder](https://mlsanity.com/paper/1702.08423/) (2017)
- [Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks](https://mlsanity.com/paper/1502.05336/) (2015)
- [Complexity of Linear Regions in Deep Networks](https://mlsanity.com/paper/1901.09021/) (2019)
- [Scaling Hamiltonian Monte Carlo Inference for Bayesian Neural Networks with Symmetric Splitting](https://mlsanity.com/paper/2010.06772/) (2020)
- [A mathematical model for automatic differentiation in machine learning](https://mlsanity.com/paper/2006.02080/) (2020)
- [Convergence of constant step stochastic gradient descent for non-smooth non-convex functions](https://mlsanity.com/paper/2005.08513/) (2020)
- [MCMC Should Mix: Learning Energy-Based Model with Neural Transport Latent Space MCMC](https://mlsanity.com/paper/2006.06897/) (2020)
- [The Geometric Foundations of Hamiltonian Monte Carlo](https://mlsanity.com/paper/1410.5110/) (2014)
- [Optimizing The Integrator Step Size for Hamiltonian Monte Carlo](https://mlsanity.com/paper/1411.6669/) (2014)
- [Towards a regularity theory for ReLU networks -- chain rule and global error estimates](https://mlsanity.com/paper/1905.04992/) (2019)
- [Bayesian Perceptron: Towards fully Bayesian Neural Networks](https://mlsanity.com/paper/2009.01730/) (2020)

## Built on by (most cited first, 1 of 1)

- [Numerical Generalized Randomized Hamiltonian Monte Carlo for piecewise smooth target densities](https://mlsanity.com/paper/2504.18210/) (2025)

## Abstract (arXiv)

We analyze the error rates of the Hamiltonian Monte Carlo algorithm with leapfrog integrator for Bayesian neural network inference. We show that due to the non-differentiability of activation functions in the ReLU family, leapfrog HMC for networks with these activation functions has a large local error rate of Omega(ε) rather than the classical error rate of O(ε3). This leads to a higher rejection rate of the proposals, making the method inefficient. We then verify our theoretical findings through empirical simulations as well as experiments on a real-world dataset that highlight the inefficiency of HMC inference on ReLU-based neural networks compared to analytical networks.