mlsanityData to 14 Oct 2026

Distributional simplicity bias and effective convexity in Energy Based Models

arXiv 2605.07844 ↗First posted 8 May 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Energy-based learning is a powerful framework for generative modelling, but its training is inherently non-convex, leading potentially to sensitivity to initialisation, poor local optima, and unstable gradient dynamics. We present a dynamical analysis of energy-based learning through the lens of the effective model, which can be interpreted as either a generalised Ising model with higher-order interactions or the Fourier expansion of the energy. Under sufficient expressivity, we show that the gradient flow induced by learning strictly positive distributions over binary variables admits two types of fixed points: data-consistent points, which exactly reproduce the target distribution, and spurious points, which satisfy stationarity without matching the target distribution. Around data-consistent points, we show that perturbations are either stable or neutral, with neutral directions leaving the effective model invariant. Finally, we show that gradient dynamics induce a hierarchy in which lower-order interactions are learned before higher-order ones. This provides a mechanistic explanation for the distributional simplicity bias and clarifies why fixed points that are not data-consistent at low orders are not observed in practice.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 34 references that are in the map. First posted on arXiv 8 May 2026.

Builds on

22 of 34 references.

References · 22

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  3. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  4. 2014The Loss Surfaces of Multilayer Networkscited by 280
  5. 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 243
  6. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 204
  7. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  8. 2018Deep learning generalizes because the parameter-function map is biased towards simple functionscited by 183
  9. 2019On the Anatomy of MCMC-Based Maximum Likelihood Learning of Energy-Based Modelscited by 105
  10. 2022Neural networks trained with SGD learn distributions of increasing complexitycited by 50
  11. 2024Neural Networks Learn Statistics of Increasing Complexitycited by 31
  12. 2025How Compositional Generalization and Creativity Improve as Diffusion Models are Trainedcited by 25
10 more
  1. 2024A distributional simplicity bias in the learning dynamics of transformerscited by 24
  2. 2024How transformers learn structured data: insights from hierarchical filteringcited by 18
  3. 2023Explaining the effects of non-convergent sampling in the training of Energy-Based Modelscited by 12
  4. 2024Fast training and sampling of Restricted Boltzmann Machinescited by 9
  5. 2023Inferring effective couplings with Restricted Boltzmann Machinescited by 8
  6. 2025Inferring Higher-Order Couplings with Neural Networkscited by 7
  7. 2024Compression of Structured Data with Autoencoders: Provable Benefit of Nonlinearities and Depthcited by 5
  8. 2026A theory of learning data statistics in diffusion models, from easy to hardcited by 5
  9. 2014Expressive Power and Approximation Errors of Restricted Boltzmann Machinescited by 4
  10. 2026Inferring effective interactions and task-related brain states from large-scale neural activity with Restricted Boltzmann Machinescited by 1

Built on by

Not cited yet.