# Training Deep Energy-Based Models with f-Divergence Minimization

arXiv: https://arxiv.org/abs/2003.03463 · HTML: https://mlsanity.com/paper/2003.03463/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Mar 2020.

## Built on by (most cited first, 29 of 29)

- [How to Train Your Energy-Based Models](https://mlsanity.com/paper/2101.03288/) (2021)
- [Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models](https://mlsanity.com/paper/2103.04922/) (2021)
- [Generative Flow Networks for Discrete Probabilistic Modeling](https://mlsanity.com/paper/2202.01361/) (2022)
- [One-step Diffusion Models with f-Divergence Distribution Matching](https://mlsanity.com/paper/2502.15681/) (2025)
- [f-GAIL: Learning f-Divergence for Generative Adversarial Imitation Learning](https://mlsanity.com/paper/2010.01207/) (2020)
- [A General Recipe for Likelihood-free Bayesian Optimization](https://mlsanity.com/paper/2206.13035/) (2022)
- [Guiding Energy-based Models via Contrastive Latent Variables](https://mlsanity.com/paper/2303.03023/) (2023)
- [Energy-Based Models for Anomaly Detection: A Manifold Diffusion Recovery Approach](https://mlsanity.com/paper/2310.18677/) (2023)
- [Analyzing and Improving the Optimization Landscape of Noise-Contrastive Estimation](https://mlsanity.com/paper/2110.11271/) (2021)
- [Sinkhorn Distance Minimization for Knowledge Distillation](https://mlsanity.com/paper/2402.17110/) (2024)
- [Preference Optimization by Estimating the Ratio of the Data Distribution](https://mlsanity.com/paper/2505.19601/) (2025)
- [f-Divergence Based Classification: Beyond the Use of Cross-Entropy](https://mlsanity.com/paper/2401.01268/) (2024)
- [Pseudo-Spherical Contrastive Divergence](https://mlsanity.com/paper/2111.00780/) (2021)
- [Bi-level Doubly Variational Learning for Energy-based Latent Variable Models](https://mlsanity.com/paper/2203.14702/) (2022)
- [M-EBM: Towards Understanding the Manifolds of Energy-Based Models](https://mlsanity.com/paper/2303.04343/) (2023)
- [Learning Unnormalized Statistical Models via Compositional Optimization](https://mlsanity.com/paper/2306.07485/) (2023)
- [Persistently Trained, Diffusion-assisted Energy-based Models](https://mlsanity.com/paper/2304.10707/) (2023)
- [POSEIDON: Physics-Optimized Seismic Energy Inference and Detection Operating Network](https://mlsanity.com/paper/2601.02264/) (2026)
- [EBMs Trained with Maximum Likelihood are Generator Models Trained with a Self-adverserial Loss](https://mlsanity.com/paper/2102.11757/) (2021)
- [Learning High-Dimensional Distributions with Latent Neural Fokker-Planck Kernels](https://mlsanity.com/paper/2105.04538/) (2021)
- [Improved Stein Variational Gradient Descent with Importance Weights](https://mlsanity.com/paper/2210.00462/) (2022)
- [A Unified Framework for Diffusion Model Unlearning with f-Divergence](https://mlsanity.com/paper/2509.21167/) (2025)
- [Generalized Munchausen Reinforcement Learning using Tsallis KL Divergence](https://mlsanity.com/paper/2301.11476/) (2023)
- [Alpha-divergence loss function for neural density ratio estimation](https://mlsanity.com/paper/2402.02041/) (2024)
- [StrEBM: A Structured Latent Energy-Based Model for Blind Source Separation](https://mlsanity.com/paper/2604.17381/) (2026)
- [EBM Life Cycle: MCMC Strategies for Synthesis, Defense, and Density Modeling](https://mlsanity.com/paper/2205.12243/) (2022)
- [Entropy-driven Unsupervised Keypoint Representation Learning in Videos](https://mlsanity.com/paper/2209.15404/) (2022)
- [On Feature Diversity in Energy-based Models](https://mlsanity.com/paper/2306.01489/) (2023)
- [STANLEY: Stochastic Gradient Anisotropic Langevin Dynamics for Learning Energy-Based Models](https://mlsanity.com/paper/2310.12667/) (2023)

## Abstract (arXiv)

Deep energy-based models (EBMs) are very flexible in distribution parametrization but computationally challenging because of the intractable partition function. They are typically trained via maximum likelihood, using contrastive divergence to approximate the gradient of the KL divergence between data and model distribution. While KL divergence has many desirable properties, other f-divergences have shown advantages in training implicit density generative models such as generative adversarial networks. In this paper, we propose a general variational framework termed f-EBM to train EBMs using any desired f-divergence. We introduce a corresponding optimization algorithm and prove its local convergence property with non-linear dynamical systems theory. Experimental results demonstrate the superiority of f-EBM over contrastive divergence, as well as the benefits of training EBMs using f-divergences other than KL.