# Log-Concave Coupling for Sampling Neural Net Posteriors

arXiv: https://arxiv.org/abs/2407.18802 · HTML: https://mlsanity.com/paper/2407.18802/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 22 references that are in the map.
- First posted on arXiv 26 Jul 2024.

## Builds on (most cited first, 5 of 5)

- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Theoretical guarantees for sampling and inference in generative models with latent diffusions](https://mlsanity.com/paper/1903.01608/) (2019)
- [Sampling with Riemannian Hamiltonian Monte Carlo in a Constrained Space](https://mlsanity.com/paper/2202.01908/) (2022)
- [Risk Bounds for High-dimensional Ridge Function Combinations Including Neural Networks](https://mlsanity.com/paper/1607.01434/) (2016)
- [Fast sampling from constrained spaces using the Metropolis-adjusted Mirror Langevin algorithm](https://mlsanity.com/paper/2312.08823/) (2023)

## Abstract (arXiv)

In this work, we present a sampling algorithm for single hidden layer neural networks. This algorithm is built upon a recursive series of Bayesian posteriors using a method we call Greedy Bayes. Sampling of the Bayesian posterior for neuron weight vectors w of dimension d is challenging because of its multimodality. Our algorithm to tackle this problem is based on a coupling of the posterior density for w with an auxiliary random variable xi. The resulting reverse conditional w|xi of neuron weights given auxiliary random variable is shown to be log concave. In the construction of the posterior distributions we provide some freedom in the choice of the prior. In particular, for Gaussian priors on w with suitably small variance, the resulting marginal density of the auxiliary variable xi is proven to be strictly log concave for all dimensions d. For a uniform prior on the unit ℓ1 ball, evidence is given that the density of xi is again strictly log concave for sufficiently large d. The score of the marginal density of the auxiliary random variable xi is determined by an expectation over w|xi and thus can be computed by various rapidly mixing Markov Chain Monte Carlo methods. Moreover, the computation of the score of xi permits methods of sampling xi by a stochastic diffusion (Langevin dynamics) with drift function built from this score. With such dynamics, information-theoretic methods pioneered by Bakry and Emery show that accurate sampling of xi is obtained rapidly when its density is indeed strictly log-concave. After which, one more draw from w|xi, produces neuron weights w whose marginal distribution is from the desired posterior.