# Training Energy-Based Models with Diffusion Contrastive Divergences

arXiv: https://arxiv.org/abs/2307.01668 · HTML: https://mlsanity.com/paper/2307.01668/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 55 references that are in the map.
- First posted on arXiv 4 Jul 2023.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [Improved Techniques for Training Score-Based Generative Models](https://mlsanity.com/paper/2006.09011/) (2020)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [Maximum Likelihood Training of Score-Based Diffusion Models](https://mlsanity.com/paper/2101.09258/) (2021)
- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models](https://mlsanity.com/paper/1810.01367/) (2018)
- [Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One](https://mlsanity.com/paper/1912.03263/) (2019)
- [Sliced Score Matching: A Scalable Approach to Density and Score Estimation](https://mlsanity.com/paper/1905.07088/) (2019)
- [How to Train Your Energy-Based Models](https://mlsanity.com/paper/2101.03288/) (2021)
- [Residual Flows for Invertible Generative Modeling](https://mlsanity.com/paper/1906.02735/) (2019)
- [A Theory of Generative ConvNet](https://mlsanity.com/paper/1602.03264/) (2016)
- [Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model](https://mlsanity.com/paper/1904.09770/) (2019)
- [Improved Contrastive Divergence Training of Energy Based Models](https://mlsanity.com/paper/2012.01316/) (2020)
- [Learning Energy-Based Models by Diffusion Recovery Likelihood](https://mlsanity.com/paper/2012.08125/) (2020)
- [Deep Structured Energy Based Models for Anomaly Detection](https://mlsanity.com/paper/1605.07717/) (2016)
- [Generalized Denoising Auto-Encoders as Generative Models](https://mlsanity.com/paper/1305.6663/) (2013)
- [Flow Contrastive Estimation of Energy-Based Models](https://mlsanity.com/paper/1912.00589/) (2019)
- [Generalized Energy Based Models](https://mlsanity.com/paper/2003.05033/) (2020)
- [No MCMC for me: Amortized sampling for fast and stable training of energy-based models](https://mlsanity.com/paper/2010.04230/) (2020)
- [Learning Descriptor Networks for 3D Shape Synthesis and Analysis](https://mlsanity.com/paper/1804.00586/) (2018)
- [A Tale of Two Flows: Cooperative Learning of Langevin Flow and Normalizing Flow Toward Energy-Based Model](https://mlsanity.com/paper/2205.06924/) (2022)

## Built on by (most cited first, 9 of 9)

- [One-Step Diffusion Distillation through Score Implicit Matching](https://mlsanity.com/paper/2410.16794/) (2024)
- [Diff-Instruct++: Training One-step Text-to-image Generator Model to Align with Human Preferences](https://mlsanity.com/paper/2410.18881/) (2024)
- [Flow Generator Matching](https://mlsanity.com/paper/2410.19310/) (2024)
- [TEA: Test-time Energy Adaptation](https://mlsanity.com/paper/2311.14402/) (2023)
- [Energy Discrepancies: A Score-Independent Loss for Energy-Based Models](https://mlsanity.com/paper/2307.06431/) (2023)
- [David and Goliath: Small One-step Model Beats Large Diffusion with Score Post-training](https://mlsanity.com/paper/2410.20898/) (2024)
- [MITA: Bridging the Gap between Model and Data for Test-time Adaptation](https://mlsanity.com/paper/2410.09398/) (2024)
- [Conditional Denoising Model as a Physical Surrogate Model](https://mlsanity.com/paper/2601.21021/) (2026)
- [ZeroDiff++: Substantial Unseen Visual-semantic Correlation in Zero-shot Learning](https://mlsanity.com/paper/2602.12401/) (2026)

## Abstract (arXiv)

Energy-Based Models (EBMs) have been widely used for generative modeling. Contrastive Divergence (CD), a prevailing training objective for EBMs, requires sampling from the EBM with Markov Chain Monte Carlo methods (MCMCs), which leads to an irreconcilable trade-off between the computational burden and the validity of the CD. Running MCMCs till convergence is computationally intensive. On the other hand, short-run MCMC brings in an extra non-negligible parameter gradient term that is difficult to handle. In this paper, we provide a general interpretation of CD, viewing it as a special instance of our proposed Diffusion Contrastive Divergence (DCD) family. By replacing the Langevin dynamic used in CD with other EBM-parameter-free diffusion processes, we propose a more efficient divergence. We show that the proposed DCDs are both more computationally efficient than the CD and are not limited to a non-negligible gradient term. We conduct intensive experiments, including both synthesis data modeling and high-dimensional image denoising and generation, to show the advantages of the proposed DCDs. On the synthetic data learning and image denoising experiments, our proposed DCD outperforms CD by a large margin. In image generation experiments, the proposed DCD is capable of training an energy-based model for generating the Celab-A 32× 32 dataset, which is comparable to existing EBMs.