# Neural Empirical Bayes

arXiv: https://arxiv.org/abs/1903.02334 · HTML: https://mlsanity.com/paper/1903.02334/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Mar 2019.

## Built on by (most cited first, 30 of 46)

- [Improved Techniques for Training Score-Based Generative Models](https://mlsanity.com/paper/2006.09011/) (2020)
- [PhysDiff: Physics-Guided Human Motion Diffusion Model](https://mlsanity.com/paper/2212.02500/) (2022)
- [Unlocking Guidance for Discrete State-Space Diffusion and Flow Models](https://mlsanity.com/paper/2406.01572/) (2024)
- [Gradient Step Denoiser for convergent Plug-and-Play](https://mlsanity.com/paper/2110.03220/) (2021)
- [Generative Skill Chaining: Long-Horizon Skill Planning with Diffusion Models](https://mlsanity.com/paper/2401.03360/) (2023)
- [Generative Time Series Forecasting with Diffusion, Denoise, and Disentanglement](https://mlsanity.com/paper/2301.03028/) (2023)
- [Diffusion Models for Time Series Applications: A Survey](https://mlsanity.com/paper/2305.00624/) (2023)
- [Estimating High Order Gradients of the Data Distribution by Denoising](https://mlsanity.com/paper/2111.04726/) (2021)
- [Energy-Based Imitation Learning](https://mlsanity.com/paper/2004.09395/) (2020)
- [3D molecule generation by denoising voxel grids](https://mlsanity.com/paper/2306.07473/) (2023)
- [Protein Discovery with Discrete Walk-Jump Sampling](https://mlsanity.com/paper/2306.12360/) (2023)
- [Probabilistic Adaptation of Text-to-Video Models](https://mlsanity.com/paper/2306.01872/) (2023)
- [Stochastic Localization via Iterative Posterior Sampling](https://mlsanity.com/paper/2402.10758/) (2024)
- [Diffusion Models in textitDe Novo Drug Design](https://mlsanity.com/paper/2406.08511/) (2024)
- [The Rise of Diffusion Models in Time-Series Forecasting](https://mlsanity.com/paper/2401.03006/) (2024)
- [A Geometric Explanation of the Likelihood OOD Detection Paradox](https://mlsanity.com/paper/2403.18910/) (2024)
- [Structure-based drug design by denoising voxel grids](https://mlsanity.com/paper/2405.03961/) (2024)
- [Improved Autoregressive Modeling with Distribution Smoothing](https://mlsanity.com/paper/2103.15089/) (2021)
- [A Geometric Perspective on Diffusion Models](https://mlsanity.com/paper/2305.19947/) (2023)
- [Score-based Denoising Diffusion with Non-Isotropic Gaussian Noise Models](https://mlsanity.com/paper/2210.12254/) (2022)
- [Chain of Log-Concave Markov Chains](https://mlsanity.com/paper/2305.19473/) (2023)
- [Diffusion Variational Autoencoder for Tackling Stochasticity in Multi-Step Regression Stock Price Prediction](https://mlsanity.com/paper/2309.00073/) (2023)
- [Multimeasurement Generative Models](https://mlsanity.com/paper/2112.09822/) (2021)
- [A Comprehensive Survey on Generative Diffusion Models for Structured Data](https://mlsanity.com/paper/2306.04139/) (2023)
- [Denoising Deep Generative Models](https://mlsanity.com/paper/2212.01265/) (2022)
- [Universal Smoothed Score Functions for Generative Modeling](https://mlsanity.com/paper/2303.11669/) (2023)
- [Denoising Diffusions with Optimal Transport: Localization, Curvature, and Multi-Scale Complexity](https://mlsanity.com/paper/2411.01629/) (2024)
- [JAMUN: Bridging Smoothed Molecular Dynamics and Score-Based Learning for Conformational Ensembles](https://mlsanity.com/paper/2410.14621/) (2024)
- [Optimal Denoising in Score-Based Generative Models: The Role of Data Regularity](https://mlsanity.com/paper/2503.12966/) (2025)
- [Geometric Regularity in Deterministic Sampling Dynamics of Diffusion-based Generative Models](https://mlsanity.com/paper/2506.10177/) (2025)

## Abstract (arXiv)

We unify textitkernel density estimation and textitempirical Bayes and address a set of problems in unsupervised learning with a geometric interpretation of those methods, rooted in the textitconcentration of measure phenomenon. Kernel density is viewed symbolically as Xrightharpoonup Y where the random variable X is smoothed to Y= X+N(0,σ2 Id), and empirical Bayes is the machinery to denoise in a least-squares sense, which we express as X leftharpoondown Y. A learning objective is derived by combining these two, symbolically captured by X rightleftharpoons Y. Crucially, instead of using the original nonparametric estimators, we parametrize textitthe energy function with a neural network denoted by φ; at optimality, nabla φ ≈ -nabla log f where f is the density of Y. The optimization problem is abstracted as interactions of high-dimensional spheres which emerge due to the concentration of isotropic gaussians. We introduce two algorithmic frameworks based on this machinery: (i) a "walk-jump" sampling scheme that combines Langevin MCMC (walks) and empirical Bayes (jumps), and (ii) a probabilistic framework for textitassociative memory, called NEBULA, defined `a la Hopfield by the textitgradient flow of the learned energy to a set of attractors. We finish the paper by reporting the emergence of very rich "creative memories" as attractors of NEBULA for highly-overlapping spheres.