mlsanityData to 14 Oct 2026

Energy Discrepancies: A Score-Independent Loss for Energy-Based Models

arXiv 2307.06431 ↗First posted 12 Jul 2023stat.ML cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Energy-based models are a simple yet powerful class of probabilistic models, but their widespread adoption has been limited by the computational burden of training them. We propose a novel loss function called Energy Discrepancy (ED) which does not rely on the computation of scores or expensive Markov chain Monte Carlo. We show that ED approaches the explicit score matching and negative log-likelihood loss under different limits, effectively interpolating between both. Consequently, minimum ED estimation overcomes the problem of nearsightedness encountered in score-based estimation methods, while also enjoying theoretical guarantees. Through numerical experiments, we demonstrate that ED learns low-dimensional data distributions faster and more accurately than explicit score matching or contrastive divergence. For high-dimensional image data, we describe how the manifold hypothesis puts limitations on our approach and demonstrate the effectiveness of energy discrepancy by training the energy-based model as a prior of a variational decoder model.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 42 of its 71 references that are in the map. First posted on arXiv 12 Jul 2023.

Builds on

42 of 71 references.

References · 42

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  7. 2013Auto-Encoding Variational Bayescited by 4,053
  8. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  9. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,545
  10. 2018Computational Optimal Transportcited by 780
  11. 2021Maximum Likelihood Training of Score-Based Diffusion Modelscited by 672
  12. 2019Your Classifier is Secretly an Energy Based Model and You Should Treat it Like Onecited by 405
30 more
  1. 2019Sliced Score Matching: A Scalable Approach to Density and Score Estimationcited by 384
  2. 2021How to Train Your Energy-Based Modelscited by 274
  3. 2016A Kernelized Stein Discrepancy for Goodness-of-fit Tests and Model Evaluationcited by 236
  4. 2019Diagnosing and Enhancing VAE Modelscited by 193
  5. 2016A Theory of Generative ConvNetcited by 174
  6. 2019Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Modelcited by 148
  7. 2016A Kernel Test of Goodness of Fitcited by 143
  8. 2019From Variational to Deterministic Autoencoderscited by 133
  9. 2020Improved Contrastive Divergence Training of Energy Based Modelscited by 130
  10. 2020Learning Energy-Based Models by Diffusion Recovery Likelihoodcited by 123
  11. 2018Efficient GAN-Based Anomaly Detectioncited by 122
  12. 2022Score-Based Generative Models Detect Manifoldscited by 122
  13. 2017Measuring Sample Quality with Kernelscited by 119
  14. 2019On the Anatomy of MCMC-Based Maximum Likelihood Learning of Energy-Based Modelscited by 105
  15. 2020Learning Latent Space Energy-Based Prior Modelcited by 104
  16. 2021Oops I Took A Gradient: Scalable Sampling for Discrete Distributionscited by 92
  17. 2020Generalized Energy Based Modelscited by 73
  18. 2019Minimum Stein Discrepancy Estimatorscited by 57
  19. 2019Maximum Entropy Generators for Energy-Based Modelscited by 56
  20. 2020Learning Discrete Energy-based Models via Auxiliary-variable Local Explorationcited by 30
  21. 2019Learning Energy-Based Models in High-Dimensional Spaces with Multi-scale Denoising Score Matchingcited by 26
  22. 2018Conditional Noise-Contrastive Estimation of Unnormalised Modelscited by 25
  23. 2018Spread Divergencecited by 23
  24. 2022Adaptive Multi-stage Density Ratio Estimation for Learning Latent Space Energy-based Modelcited by 18
  25. 2022Towards Healing the Blindness of Score Matchingcited by 13
  26. 2022Learning Neural Set Functions Under the Optimal Subset Oraclecited by 10
  27. 2023Training Energy-Based Models with Diffusion Contrastive Divergencescited by 9
  28. 2020Contrastive Divergence Learning is a Time Reversal Adversarial Gamecited by 7
  29. 2014Generative Modeling of Convolutional Neural Networkscited by 5
  30. 2023Using Perturbation to Improve Goodness-of-Fit Tests based on Kernelized Stein Discrepancycited by 4

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 3; 2025: 5.

9 papers.

  1. 2025Discrete Neural Flow Samplers with Locally Equivariant Transformercited by 13
  2. 2025Follow the Energy, Find the Path: Riemannian Metrics from Energy-Based Modelscited by 7
  3. 2024Energy-Based Modelling for Discrete and Mixed Data via Heat Equations on Structured Spacescited by 5
  4. 2025TabRep: Training Tabular Diffusion Models with a Simple and Effective Continuous Representationcited by 5
  5. 2025Learning Latent Energy-Based Models via Interacting Particle Langevin Dynamicscited by 5
  6. 2023Training Discrete Energy-Based Models with Energy Discrepancycited by 0
  7. 2024Inverse Entropic Optimal Transport Solves Semi-supervised Learning via Data Likelihood Maximizationcited by 0
  8. 2024Deep Optimal Sensor Placement for Black Box Stochastic Simulationscited by 0
  9. 2025Learning Energy-Based Models by Self-normalising the Likelihoodcited by 0