# Your Diffusion Model is Secretly a Noise Classifier and Benefits from Contrastive Training

arXiv: https://arxiv.org/abs/2407.08946 · HTML: https://mlsanity.com/paper/2407.08946/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 32 of its 43 references that are in the map.
- First posted on arXiv 12 Jul 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 32)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [StarGAN v2: Diverse Image Synthesis for Multiple Domains](https://mlsanity.com/paper/1912.01865/) (2019)
- [Improved Techniques for Training Score-Based Generative Models](https://mlsanity.com/paper/2006.09011/) (2020)
- [DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models](https://mlsanity.com/paper/2211.01095/) (2022)
- [On Distillation of Guided Diffusion Models](https://mlsanity.com/paper/2210.03142/) (2022)
- [Pseudo Numerical Methods for Diffusion Models on Manifolds](https://mlsanity.com/paper/2202.09778/) (2022)
- [Tackling the Generative Learning Trilemma with Denoising Diffusion GANs](https://mlsanity.com/paper/2112.07804/) (2021)
- [Sliced Score Matching: A Scalable Approach to Density and Score Estimation](https://mlsanity.com/paper/1905.07088/) (2019)
- [Gotta Go Fast When Generating Data with Score-Based Models](https://mlsanity.com/paper/2105.14080/) (2021)
- [A Variational Perspective on Diffusion-Based Generative Models and Score Matching](https://mlsanity.com/paper/2106.02808/) (2021)
- [Learning Fast Samplers for Diffusion Models by Differentiating Through Sample Quality](https://mlsanity.com/paper/2202.05830/) (2022)
- [Parallel Sampling of Diffusion Models](https://mlsanity.com/paper/2305.16317/) (2023)
- [Telescoping Density-Ratio Estimation](https://mlsanity.com/paper/2006.12204/) (2020)
- [CoDi: Co-evolving Contrastive Diffusion Models for Mixed-type Tabular Synthesis](https://mlsanity.com/paper/2304.12654/) (2023)
- [Diffusion Normalizing Flow](https://mlsanity.com/paper/2110.07579/) (2021)
- [Density Ratio Estimation via Infinitesimal Classification](https://mlsanity.com/paper/2111.11010/) (2021)
- [Zero-Shot Contrastive Loss for Text-Guided Diffusion Image Style Transfer](https://mlsanity.com/paper/2303.08622/) (2023)
- [Information-Theoretic Diffusion](https://mlsanity.com/paper/2302.03792/) (2023)

## Built on by (most cited first, 5 of 5)

- [Maximum Entropy Reinforcement Learning with Diffusion Policy](https://mlsanity.com/paper/2502.11612/) (2025)
- [Diffusion and Flow-based Copulas: Forgetting and Remembering Dependencies](https://mlsanity.com/paper/2509.19707/) (2025)
- [Discrete Stochastic Localization for Non-autoregressive Generation](https://mlsanity.com/paper/2602.16169/) (2026)
- [Channel-aware Contrastive Conditional Diffusion for Multivariate Probabilistic Time Series Forecasting](https://mlsanity.com/paper/2410.02168/) (2024)
- [Discrete Stochastic Localization for Non-autoregressive Generation](https://mlsanity.com/paper/2605.12836/) (2026)

## Abstract (arXiv)

Diffusion models learn to denoise data and the trained denoiser is then used to generate new samples from the data distribution. In this paper, we revisit the diffusion sampling process and identify a fundamental cause of sample quality degradation: the denoiser is poorly estimated in regions that are far Outside Of the training Distribution (OOD), and the sampling process inevitably evaluates in these OOD regions. This can become problematic for all sampling methods, especially when we move to parallel sampling which requires us to initialize and update the entire sample trajectory of dynamics in parallel, leading to many OOD evaluations. To address this problem, we introduce a new self-supervised training objective that differentiates the levels of noise added to a sample, leading to improved OOD denoising performance. The approach is based on our observation that diffusion models implicitly define a log-likelihood ratio that distinguishes distributions with different amounts of noise, and this expression depends on denoiser performance outside the standard training distribution. We show by diverse experiments that the proposed contrastive diffusion training is effective for both sequential and parallel settings, and it improves the performance and speed of parallel samplers significantly.