mlsanityData to 14 Oct 2026

Palette: Image-to-Image Diffusion Models

arXiv 2111.05826 ↗First posted 10 Nov 2021cs.CV cs.LGcited by 1,300

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

This paper develops a unified framework for image-to-image translation based on conditional diffusion models and evaluates this framework on four challenging image-to-image translation tasks, namely colorization, inpainting, uncropping, and JPEG restoration. Our simple implementation of image-to-image diffusion models outperforms strong GAN and regression baselines on all tasks, without task-specific hyper-parameter tuning, architecture customization, or any auxiliary loss or sophisticated new techniques needed. We uncover the impact of an L2 vs. L1 loss in the denoising diffusion objective on sample diversity, and demonstrate the importance of self-attention in the neural architecture through empirical studies. Importantly, we advocate a unified evaluation protocol based on ImageNet, with human evaluation and sample quality scores (FID, Inception Score, Classification Accuracy of a pre-trained ResNet-50, and Perceptual Distance against original images). We expect this standardized evaluation protocol to play a role in advancing image-to-image translation research. Finally, we show that a generalist, multi-task diffusion model performs as well or better than task-specific specialist counterparts. Check out https://diffusion-palette.github.io for an overview of the results.

Key facts. Cited by 1,300 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 72 of its 115 references that are in the map. First posted on arXiv 10 Nov 2021.

Builds on

72 of 115 references.

References · 72

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  6. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  7. 2013Auto-Encoding Variational Bayescited by 4,053
  8. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  9. 2017Improved Training of Wasserstein GANscited by 2,235
  10. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,161
  11. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,054
  12. 2016Density estimation using Real NVPcited by 2,042
38 more
  1. 2016Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Networkcited by 1,908
  2. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  3. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,545
  4. 2021Image Super-Resolution via Iterative Refinementcited by 1,511
  5. 2016Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoisingcited by 1,358
  6. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,354
  7. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,163
  8. 2021Variational Diffusion Modelscited by 1,153
  9. 2016Colorful Image Colorizationcited by 1,114
  10. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,043
  11. 2017StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translationcited by 1,039
  12. 2016Conditional Image Generation with PixelCNN Decoderscited by 862
  13. 2018Image Transformercited by 772
  14. 2017Wasserstein GANcited by 639
  15. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 637
  16. 2021Score-based Generative Modeling in Latent Spacecited by 636
  17. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 622
  18. 2018Free-Form Image Inpainting with Gated Convolutioncited by 594
  19. 2018Generative Image Inpainting with Contextual Attentioncited by 580
  20. 2018Image Inpainting for Irregular Holes Using Partial Convolutionscited by 505
  21. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 417
  22. 2020PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Modelscited by 358
  23. 2015Deeply-Recursive Convolutional Network for Image Super-Resolutioncited by 317
  24. 2016Learning Representations for Automatic Colorizationcited by 251
  25. 2020Learning Gradient Fields for Shape Generationcited by 245
  26. 2016Generating Images with Perceptual Similarity Metrics based on Deep Networkscited by 222
  27. 2019Few-Shot Unsupervised Image-to-Image Translationcited by 222
  28. 2021Large Scale Image Completion via Co-Modulated Generative Adversarial Networkscited by 220
  29. 2021Scaling Local Self-Attention for Parameter Efficient Visual Backbonescited by 220
  30. 2021Gotta Go Fast When Generating Data with Score-Based Modelscited by 219
  31. 2016Unrolled Generative Adversarial Networkscited by 208
  32. 2018Multi-level Wavelet-CNN for Image Restorationcited by 184
  33. 2019Pluralistic Image Completioncited by 162
  34. 2019Guided Image Generation with Conditional Invertible Neural Networkscited by 147
  35. 2015Compression Artifacts Reduction by a Deep Convolutional Networkcited by 136
  36. 2019Classification Accuracy Score for Conditional Generative Modelscited by 136
  37. 2021UNIT-DDPM: UNpaired Image Translation with Denoising Diffusion Probabilistic Modelscited by 133
  38. 2021Learning to Efficiently Sample from Diffusion Probabilistic Modelscited by 129

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 126; 2023: 434; 2024: 377; 2025: 238; 2026: 122.

1,300 papers.

Sort:

By area:

  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  3. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  4. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  5. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  6. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  7. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  8. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  9. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  10. 2022Video Diffusion Modelscited by 2,305
  11. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  12. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
38 more
  1. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,430
  2. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,178
  3. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,114
  4. 2022Human Motion Diffusion Modelcited by 1,080
  5. 2022Diffusion Models in Vision: A Surveycited by 1,015
  6. 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 1,006
  7. 2022Denoising Diffusion Restoration Modelscited by 963
  8. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 938
  9. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 931
  10. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 906
  11. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 781
  12. 2022DiffEdit: Diffusion-based semantic image editing with mask guidancecited by 647
  13. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 624
  14. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 590
  15. 2022Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Modelcited by 583
  16. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 546
  17. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generationcited by 536
  18. 2023Zero-shot Image-to-Image Translationcited by 505
  19. 2022Paint by Example: Exemplar-based Image Editing with Diffusion Modelscited by 489
  20. 2023Exploiting Diffusion Prior for Real-World Image Super-Resolutioncited by 489
  21. 2022Diffusion Models for Adversarial Purificationcited by 476
  22. 2023Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusioncited by 459
  23. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 455
  24. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Modelscited by 450
  25. 2022Blended Latent Diffusioncited by 432
  26. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 379
  27. 2023ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shiftingcited by 365
  28. 2023DeepCache: Accelerating Diffusion Models for Freecited by 359
  29. 2023Human Motion Diffusion as a Generative Priorcited by 357
  30. 2023Denoising Diffusion Models for Plug-and-Play Image Restorationcited by 340
  31. 2024OminiControl: Minimal and Universal Control for Diffusion Transformercited by 340
  32. 2023Synthetic Data from Diffusion Models Improves ImageNet Classificationcited by 327
  33. 2023Repurposing Diffusion-Based Image Generators for Monocular Depth Estimationcited by 327
  34. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  35. 2023Pix2Video: Video Editing using Image Diffusioncited by 320
  36. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolationcited by 309
  37. 2023Diffusion Self-Guidance for Controllable Image Generationcited by 305
  38. 2023Understanding Diffusion Objectives as the ELBO with Simple Data Augmentationcited by 304

50 most cited of 1,300 All 1,300 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Diffusion-Based Image-to-Image Translation by Noise Correction via Prompt Interpolation
  2. 2025CycleDiff: Cycle Diffusion Models for Unpaired Image-to-image Translation
  3. 2024Distilling Diffusion Models into Conditional GANs
  4. 2023Improving Diffusion-based Image Translation using Asymmetric Gradient Guidance
  5. 2023Conditional Image Generation with Pretrained Generative Model
  6. 2025Image-to-Image Translation with Diffusion Transformers and CLIP-Based Image Conditioning
  7. 2024Test-time Conditional Text-to-Image Synthesis Using Diffusion Models
  8. 2024Simpler Diffusion (SiD2): 1.5 FID on ImageNet512 with pixel-space diffusion