mlsanityData to 7 Oct 2026

On Energy-Based Models with Overparametrized Shallow Neural Networks

arXiv 2104.07531 ↗First posted 15 Apr 2021cs.LG stat.MLcited by 8

Not on the map. Why?

Abstract (arXiv)

Energy-based models (EBMs) are a simple yet powerful framework for generative modeling. They are based on a trainable energy function which defines an associated Gibbs measure, and they can be trained and sampled from via well-established statistical tools, such as MCMC. Neural networks may be used as energy function approximators, providing both a rich class of expressive models as well as a flexible device to incorporate data structure. In this work we focus on shallow neural networks. Building from the incipient theory of overparametrized neural networks, we show that models trained in the so-called "active" regime provide a statistical advantage over their associated "lazy" or kernel regime, leading to improved adaptivity to hidden low-dimensional structure in the data distribution, as already observed in supervised learning. Our study covers both maximum likelihood and Stein Discrepancy estimators, and we validate our theoretical results with numerical experiments on synthetic data.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 28 of its 65 references that are in the map. First posted on arXiv 15 Apr 2021.

Builds on

28 of 65 references.

References · 28

  1. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  2. 2013Auto-Encoding Variational Bayescited by 3,980
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2016Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithmcited by 498
  5. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  6. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  7. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  8. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  9. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  10. 2021How to Train Your Energy-Based Modelscited by 271
  11. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  12. 2016A Kernelized Stein Discrepancy for Goodness-of-fit Tests and Model Evaluationcited by 236
16 more
  1. 2016A Theory of Generative ConvNetcited by 174
  2. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  3. 2016A Kernel Test of Goodness of Fitcited by 143
  4. 2015On the Equivalence between Kernel Quadrature Rules and Random Feature Expansionscited by 136
  5. 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
  6. 2017Measuring Sample Quality with Kernelscited by 119
  7. 2020Generative Modeling with Denoising Auto-Encoders and Langevin Samplingcited by 118
  8. 2019A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Casecited by 113
  9. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  10. 2019How do infinite width bounded norm networks look in function space?cited by 105
  11. 2015Measuring Sample Quality with Stein's Methodcited by 94
  12. 2019Limitations of Lazy Training of Two-layers Neural Networkscited by 87
  13. 2019Minimum Stein Discrepancy Estimatorscited by 57
  14. 2019Gradient Dynamics of Shallow Univariate ReLU Networkscited by 46
  15. 2021Quantifying the Benefit of Using Differentiable Learning over Tangent Kernelscited by 41
  16. 2018Nonparametric Density Estimation under Adversarial Lossescited by 26

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 0; 2023: 2; 2024: 2; 2025: 1.

8 papers.

  1. 2021Two-layer neural networks with values in a Banach spacecited by 26
  2. 2023Efficient Training of Energy-Based Models Using Jarzynski Equalitycited by 15
  3. 2024Score-based generative models break the curse of dimensionality in learning a family of sub-Gaussian probability distributionscited by 12
  4. 2024Shallow diffusion networks provably learn hidden low-dimensional structurecited by 12
  5. 2021Dual Training of Energy-Based Models with Overparametrized Shallow Neural Networkscited by 8
  6. 2023Conditionally Strongly Log-Concave Generative Modelscited by 2
  7. 2025Generalizable Reasoning through Compositional Energy Minimizationcited by 2
  8. 2021Separation Results between Fixed-Kernel and Feature-Learning Probability Metricscited by 1