# Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling

arXiv: https://arxiv.org/abs/2003.06060 · HTML: https://mlsanity.com/paper/2003.06060/

## Key facts

- Cited by 71 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Mar 2020.

## Built on by (most cited first, 30 of 71)

- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [Score-based Generative Modeling in Latent Space](https://mlsanity.com/paper/2106.05931/) (2021)
- [Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models](https://mlsanity.com/paper/2103.04922/) (2021)
- [Self-Consuming Generative Models Go MAD](https://mlsanity.com/paper/2307.01850/) (2023)
- [DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latents](https://mlsanity.com/paper/2201.00308/) (2022)
- [Refining Generative Process with Discriminator Guidance in Score-based Diffusion Models](https://mlsanity.com/paper/2211.17091/) (2022)
- [Controllable and Compositional Generation with Latent-Space Energy-Based Models](https://mlsanity.com/paper/2110.10873/) (2021)
- [Rebooting ACGAN: Auxiliary Classifier GANs with Stable Training](https://mlsanity.com/paper/2111.01118/) (2021)
- [Energy-Based Open-World Uncertainty Modeling for Confidence Calibration](https://mlsanity.com/paper/2107.12628/) (2021)
- [Training GANs with Stronger Augmentations via Contrastive Discriminator](https://mlsanity.com/paper/2103.09742/) (2021)
- [GATSBI: Generative Adversarial Training for Simulation-Based Inference](https://mlsanity.com/paper/2203.06481/) (2022)
- [Unifying Generative Models with GFlowNets and Beyond](https://mlsanity.com/paper/2209.02606/) (2022)
- [MaGNET: Uniform Sampling from Deep Generative Network Manifolds Without Retraining](https://mlsanity.com/paper/2110.08009/) (2021)
- [Local-Global MCMC kernels: the best of both worlds](https://mlsanity.com/paper/2111.02702/) (2021)
- [Polarity Sampling: Quality and Diversity Control of Pre-Trained Generative Networks via Singular Values](https://mlsanity.com/paper/2203.01993/) (2022)
- [On Sampling with Approximate Transport Maps](https://mlsanity.com/paper/2302.04763/) (2023)
- [Unifying GANs and Score-Based Diffusion as Generative Particle Models](https://mlsanity.com/paper/2305.16150/) (2023)
- [Complexity Matters: Rethinking the Latent Space for Generative Modeling](https://mlsanity.com/paper/2307.08283/) (2023)
- [Learning Energy-Based Models by Cooperative Diffusion Recovery Likelihood](https://mlsanity.com/paper/2309.05153/) (2023)
- [Deep MMD Gradient Flow without adversarial training](https://mlsanity.com/paper/2405.06780/) (2024)
- [Outsourced diffusion sampling: Efficient posterior inference in latent spaces of generative models](https://mlsanity.com/paper/2502.06999/) (2025)
- [Learning Probabilistic Models from Generator Latent Spaces with Hat EBM](https://mlsanity.com/paper/2210.16486/) (2022)
- [Latent Space Refinement for Deep Generative Models](https://mlsanity.com/paper/2106.00792/) (2021)
- [Bounds all around: training energy-based models with bidirectional bounds](https://mlsanity.com/paper/2111.00929/) (2021)
- [BIGRoC: Boosting Image Generation via a Robust Classifier](https://mlsanity.com/paper/2108.03702/) (2021)
- [DiffRatio: Training One-Step Diffusion Models Without Teacher Supervision](https://mlsanity.com/paper/2502.08005/) (2025)
- [Preference Optimization by Estimating the Ratio of the Data Distribution](https://mlsanity.com/paper/2505.19601/) (2025)
- [Robust and Controllable Object-Centric Learning through Energy-based Models](https://mlsanity.com/paper/2210.05519/) (2022)
- [Bring Metric Functions into Diffusion Models](https://mlsanity.com/paper/2401.02414/) (2024)
- [Latent reweighting, an almost free improvement for GANs](https://mlsanity.com/paper/2110.09803/) (2021)

## Abstract (arXiv)

We show that the sum of the implicit generator log-density log pg of a GAN with the logit score of the discriminator defines an energy function which yields the true data density when the generator is imperfect but the discriminator is optimal, thus making it possible to improve on the typical generator (with implicit density pg). To make that practical, we show that sampling from this modified density can be achieved by sampling in latent space according to an energy-based model induced by the sum of the latent prior log-density and the discriminator output score. This can be achieved by running a Langevin MCMC in latent space and then applying the generator function, which we call Discriminator Driven Latent Sampling~(DDLS). We show that DDLS is highly efficient compared to previous methods which work in the high-dimensional pixel space and can be applied to improve on previously trained GANs of many types. We evaluate DDLS on both synthetic and real-world datasets qualitatively and quantitatively. On CIFAR-10, DDLS substantially improves the Inception Score of an off-the-shelf pre-trained SN-GAN~citepsngan from 8.22 to 9.09 which is even comparable to the class-conditional BigGAN~citepbiggan model. This achieves a new state-of-the-art in unconditional image synthesis setting without introducing extra parameters or additional training.