Neural Empirical Bayes

We unify textitkernel density estimation and textitempirical Bayes and address a set of problems in unsupervised learning with a geometric interpretation of those methods, rooted in the textitconcentration of measure phenomenon. Kernel density is viewed symbolically as Xrightharpoonup Y where the random variable X is smoothed to Y= X+N(0,σ2 Id), and empirical Bayes is the machinery to denoise in a least-squares sense, which we express as X leftharpoondown Y. A learning objective is derived by combining these two, symbolically captured by X rightleftharpoons Y. Crucially, instead of using the original nonparametric estimators, we parametrize textitthe energy function with a neural network denoted by φ; at optimality, nabla φ ≈ -nabla log f where f is the density of Y. The optimization problem is abstracted as interactions of high-dimensional spheres which emerge due to the concentration of isotropic gaussians. We introduce two algorithmic frameworks based on this machinery: (i) a "walk-jump" sampling scheme that combines Langevin MCMC (walks) and empirical Bayes (jumps), and (ii) a probabilistic framework for textitassociative memory, called NEBULA, defined `a la Hopfield by the textitgradient flow of the learned energy to a set of attractors. We finish the paper by reporting the emergence of very rich "creative memories" as attractors of NEBULA for highly-overlapping spheres.