# D2C: Diffusion-Denoising Models for Few-shot Conditional Generation

arXiv: https://arxiv.org/abs/2106.06819 · HTML: https://mlsanity.com/paper/2106.06819/

## Key facts

- Cited by 96 papers in the map (data through 14 Oct 2026).
- Builds on 90 of its 107 references that are in the map.
- First posted on arXiv 12 Jun 2021.

## Builds on (most cited first, 30 of 90)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)

## Built on by (most cited first, 30 of 96)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [Diffusion Models in Vision: A Survey](https://mlsanity.com/paper/2209.04747/) (2022)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [On Distillation of Guided Diffusion Models](https://mlsanity.com/paper/2210.03142/) (2022)
- [AudioLDM: Text-to-Audio Generation with Latent Diffusion Models](https://mlsanity.com/paper/2301.12503/) (2023)
- [Diffusion Autoencoders: Toward a Meaningful and Decodable Representation](https://mlsanity.com/paper/2111.15640/) (2021)
- [SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery](https://mlsanity.com/paper/2207.08051/) (2022)
- [PhysDiff: Physics-Guided Human Motion Diffusion Model](https://mlsanity.com/paper/2212.02500/) (2022)
- [SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation](https://mlsanity.com/paper/2212.04493/) (2022)
- [Score-Based Generative Modeling with Critically-Damped Langevin Diffusion](https://mlsanity.com/paper/2112.07068/) (2021)
- [Diffusion models as plug-and-play priors](https://mlsanity.com/paper/2206.09012/) (2022)
- [Dual Diffusion Implicit Bridges for Image-to-Image Translation](https://mlsanity.com/paper/2203.08382/) (2022)
- [From data to functa: Your data point is a function and you can treat it like one](https://mlsanity.com/paper/2201.12204/) (2022)
- [Geometric Latent Diffusion Models for 3D Molecule Generation](https://mlsanity.com/paper/2305.01140/) (2023)
- [Flow Matching in Latent Space](https://mlsanity.com/paper/2307.08698/) (2023)
- [Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuning](https://mlsanity.com/paper/2308.06038/) (2023)
- [Concrete Score Matching: Generalized Score Matching for Discrete Data](https://mlsanity.com/paper/2211.00802/) (2022)
- [Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding](https://mlsanity.com/paper/2211.06956/) (2022)
- [Diffusion-SDF: Conditional Generative Modeling of Signed Distance Functions](https://mlsanity.com/paper/2211.13757/) (2022)
- [DiffusionSat: A Generative Foundation Model for Satellite Imagery](https://mlsanity.com/paper/2312.03606/) (2023)
- [Affordance Diffusion: Synthesizing Hand-Object Interactions](https://mlsanity.com/paper/2303.12538/) (2023)
- [DiffiT: Diffusion Vision Transformers for Image Generation](https://mlsanity.com/paper/2312.02139/) (2023)
- [TrojDiff: Trojan Attacks on Diffusion Models with Diverse Targets](https://mlsanity.com/paper/2303.05762/) (2023)
- [Diffusion Causal Models for Counterfactual Estimation](https://mlsanity.com/paper/2202.10166/) (2022)
- [Image Denoising: The Deep Learning Revolution and Beyond -- A Survey Paper --](https://mlsanity.com/paper/2301.03362/) (2023)
- [Latent Diffusion Energy-Based Model for Interpretable Text Modeling](https://mlsanity.com/paper/2206.05895/) (2022)
- [DiffPose: Multi-hypothesis Human Pose Estimation using Diffusion models](https://mlsanity.com/paper/2211.16487/) (2022)
- [Unsupervised Representation Learning from Pre-trained Diffusion Probabilistic Models](https://mlsanity.com/paper/2212.12990/) (2022)

## Abstract (arXiv)

Conditional generative models of high-dimensional images have many applications, but supervision signals from conditions to images can be expensive to acquire. This paper describes Diffusion-Decoding models with Contrastive representations (D2C), a paradigm for training unconditional variational autoencoders (VAEs) for few-shot conditional image generation. D2C uses a learned diffusion-based prior over the latent representations to improve generation and contrastive self-supervised learning to improve representation quality. D2C can adapt to novel generation tasks conditioned on labels or manipulation constraints, by learning from as few as 100 labeled examples. On conditional generation from new labels, D2C achieves superior performance over state-of-the-art VAEs and diffusion models. On conditional image manipulation, D2C generations are two orders of magnitude faster to produce over StyleGAN2 ones and are preferred by 50% - 60% of the human evaluators in a double-blind study.