mlsanityData to 7 Oct 2026

Generalized Contrastive Divergence: Joint Training of Energy-Based Model and Diffusion Model through Inverse Reinforcement Learning

arXiv 2312.03397 ↗First posted 6 Dec 2023cs.LG cs.AIcited by 0

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

We present Generalized Contrastive Divergence (GCD), a novel objective function for training an energy-based model (EBM) and a sampler simultaneously. GCD generalizes Contrastive Divergence (Hinton, 2002), a celebrated algorithm for training EBM, by replacing Markov Chain Monte Carlo (MCMC) distribution with a trainable sampler, such as a diffusion model. In GCD, the joint training of EBM and a diffusion model is formulated as a minimax problem, which reaches an equilibrium when both models converge to the data distribution. The minimax learning with GCD bears interesting equivalence to inverse reinforcement learning, where the energy corresponds to a negative reward, the diffusion model is a policy, and the real data is expert demonstrations. We present preliminary yet promising results showing that joint training is beneficial for both EBM and a diffusion model. GCD enables EBM training without MCMC while improving the sample quality of a diffusion model.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 26 of its 38 references that are in the map. First posted on arXiv 6 Dec 2023.

Builds on

26 of 38 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards29 Sep
  2. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models25 May
  3. 2023Training Diffusion Models with Reinforcement Learning22 May

Other references · 23

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  5. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  6. 2016Generative Adversarial Imitation Learningcited by 1,576
  7. 2021Maximum Likelihood Training of Score-Based Diffusion Modelscited by 670
  8. 2022Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptionscited by 440
  9. 2021How to Train Your Energy-Based Modelscited by 271
  10. 2019On Gradient Descent Ascent for Nonconvex-Concave Minimax Problemscited by 266
  11. 2023HIVE: Harnessing Human Feedback for Instructional Visual Editingcited by 172
  12. 2021On Fast Sampling of Diffusion Probabilistic Modelscited by 170
11 more
  1. 2020Improved Contrastive Divergence Training of Energy Based Modelscited by 127
  2. 2020Learning Energy-Based Models by Diffusion Recovery Likelihoodcited by 122
  3. 2020VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Modelscited by 103
  4. 2023Optimizing DDPM Sampling with Shortcut Fine-Tuningcited by 81
  5. 2022Score-based Generative Modeling Secretly Minimizes the Wasserstein Distancecited by 50
  6. 2022A Tale of Two Flows: Cooperative Learning of Langevin Flow and Normalizing Flow Toward Energy-Based Modelcited by 44
  7. 2018Divergence Triangle for Joint Training of Generator Model, Energy-based Model, and Inference Modelcited by 34
  8. 2021Autoencoding Under Normalization Constraintscited by 26
  9. 2023Energy-guided Entropic Neural Optimal Transportcited by 25
  10. 2023Guiding Energy-based Models via Contrastive Latent Variablescited by 17
  11. 2023Energy-Based Models for Anomaly Detection: A Manifold Diffusion Recovery Approachcited by 17

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Maximum Entropy Inverse Reinforcement Learning of Diffusion Models with Energy-Based Models
  2. 2026iADD: Improving Alignment and Diversity in Diffusion Policy Optimization
  3. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning
  4. 2026Reverse Flow Matching: A Unified Framework for Online Reinforcement Learning with Diffusion and Flow Policies
  5. 2020Revisiting Maximum Entropy Inverse Reinforcement Learning: New Perspectives and Algorithms
  6. 2026DiPOD: Diffusion Policy Optimization without Drifting Apart
  7. 2023Towards Controllable Diffusion Models via Reward-Guided Exploration
  8. 2024Avoiding mode collapse in diffusion models fine-tuned with reinforcement learning