Hamiltonian Monte Carlo on ReLU Neural Networks is Inefficient
Not on the map. Why?
Abstract (arXiv)
We analyze the error rates of the Hamiltonian Monte Carlo algorithm with leapfrog integrator for Bayesian neural network inference. We show that due to the non-differentiability of activation functions in the ReLU family, leapfrog HMC for networks with these activation functions has a large local error rate of Omega(ε) rather than the classical error rate of O(ε3). This leads to a higher rejection rate of the proposals, making the method inefficient. We then verify our theoretical findings through empirical simulations as well as experiments on a real-world dataset that highlight the inefficiency of HMC inference on ReLU-based neural networks compared to analytical networks.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 11 of its 33 references that are in the map. First posted on arXiv 29 Oct 2024.
Builds on
11 of 33 references.
References · 11
- 2017Age Progression/Regression by Conditional Adversarial Autoencodercited by 426
- 2015Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networkscited by 295
- 2019Complexity of Linear Regions in Deep Networkscited by 166
- 2020Scaling Hamiltonian Monte Carlo Inference for Bayesian Neural Networks with Symmetric Splittingcited by 54
- 2020A mathematical model for automatic differentiation in machine learningcited by 44
- 2020Convergence of constant step stochastic gradient descent for non-smooth non-convex functionscited by 27
- 2020MCMC Should Mix: Learning Energy-Based Model with Neural Transport Latent Space MCMCcited by 25
- 2014The Geometric Foundations of Hamiltonian Monte Carlocited by 19
- 2014Optimizing The Integrator Step Size for Hamiltonian Monte Carlocited by 8
- 2019Towards a regularity theory for ReLU networks -- chain rule and global error estimatescited by 8
- 2020Bayesian Perceptron: Towards fully Bayesian Neural Networkscited by 4
Built on by
Citing papers per year: 2025: 1.
1 paper.