mlsanityData to 7 Oct 2026

eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers

arXiv 2211.01324 ↗First posted 2 Nov 2022cs.CV cs.LGcited by 921

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Large-scale diffusion-based generative models have led to breakthroughs in text-conditioned high-resolution image synthesis. Starting from random noise, such text-to-image diffusion models gradually synthesize images in an iterative fashion while conditioning on text prompts. We find that their synthesis behavior qualitatively changes throughout this process: Early in sampling, generation strongly relies on the text prompt to generate text-aligned content, while later, the text conditioning is almost entirely ignored. This suggests that sharing model parameters throughout the entire generation process may not be ideal. Therefore, in contrast to existing works, we propose to train an ensemble of text-to-image diffusion models specialized for different synthesis stages. To maintain training efficiency, we initially train a single model, which is then split into specialized models that are trained for the specific stages of the iterative generation process. Our ensemble of diffusion models, called eDiff-I, results in improved text alignment while maintaining the same inference computation cost and preserving high visual quality, outperforming previous large-scale text-to-image diffusion models on the standard benchmark. In addition, we train our model to exploit a variety of embeddings for conditioning, including the T5 text, CLIP text, and CLIP image embeddings. We show that these different embeddings lead to different behaviors. Notably, the CLIP image embedding allows an intuitive way of transferring the style of a reference image to the target text-to-image output. Lastly, we show a technique that enables eDiff-I's "paint-with-words" capability. A user can select the word in the input text and paint it in a canvas to control the output, which is very handy for crafting the desired image in mind. The project page is available at https://deepimagination.cc/eDiff-I/

Key facts. Cited by 921 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 82 of its 91 references that are in the map. First posted on arXiv 2 Nov 2022.

Builds on

82 of 91 references.

Closest ancestors · 10

Same direction, newest first

  1. 2022DiffEdit: Diffusion-based semantic image editing with mask guidance20 Oct
  2. 2022Imagic: Text-Based Real Image Editing with Diffusion Models17 Oct
  3. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  4. 2022Blended Latent Diffusion6 Jun
  5. 2022Pretraining is All You Need for Image-to-Image Translation25 May
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  8. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  9. 2021Blended Diffusion for Text-driven Editing of Natural Images29 Nov
  10. 2021Palette: Image-to-Image Diffusion Models10 Nov

Other references · 72

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2020Denoising Diffusion Implicit Modelscited by 9,655
  9. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  10. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  11. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  12. 2020Scaling Laws for Neural Language Modelscited by 6,411
38 more
  1. 2022Classifier-Free Diffusion Guidancecited by 5,717
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  4. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  6. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  9. 2022Video Diffusion Modelscited by 2,280
  10. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  11. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  12. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  13. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  14. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  15. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  16. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  17. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  18. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  19. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  20. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  21. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,036
  22. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  23. 2021Scaling Vision Transformerscited by 993
  24. 2021Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Datacited by 961
  25. 2022Denoising Diffusion Restoration Modelscited by 951
  26. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  27. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  28. 2021Diffusion Probabilistic Models for 3D Point Cloud Generationcited by 734
  29. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 695
  30. 2021Score-based Generative Modeling in Latent Spacecited by 634
  31. 2021CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputationcited by 577
  32. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  33. 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 552
  34. 2022Fast Sampling of Diffusion Models with Exponential Integratorcited by 526
  35. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  36. 2022GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generationcited by 511
  37. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  38. 2022Improving Diffusion Models for Inverse Problems using Manifold Constraintscited by 501

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 16; 2023: 327; 2024: 317; 2025: 185; 2026: 76.

921 papers.

Sort:

By area:

  1. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  2. 2023Consistency Modelscited by 1,778
  3. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  4. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  5. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  6. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  7. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  8. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  9. 2023One-step Diffusion with Distribution Matching Distillationcited by 978
  10. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 928
  11. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  12. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
38 more
  1. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  2. 2023Adversarial Diffusion Distillationcited by 741
  3. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 696
  4. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 694
  5. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creationcited by 684
  6. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 683
  7. 2024Improved Distribution Matching Distillation for Fast Image Synthesiscited by 648
  8. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 589
  9. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  10. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  11. 2023Exploiting Diffusion Prior for Real-World Image Super-Resolutioncited by 486
  12. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  13. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 455
  14. 2024InstantID: Zero-shot Identity-Preserving Generation in Secondscited by 431
  15. 2023ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generationcited by 414
  16. 2023Simple diffusion: End-to-end diffusion for high resolution imagescited by 413
  17. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 396
  18. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priorscited by 395
  19. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  20. 2023InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generationcited by 387
  21. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
  22. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349
  23. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 343
  24. 2023Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Priorcited by 341
  25. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuningcited by 338
  26. 2024SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformerscited by 331
  27. 2023Synthetic Data from Diffusion Models Improves ImageNet Classificationcited by 326
  28. 2023ControlVideo: Training-free Controllable Text-to-Video Generationcited by 324
  29. 2023Pix2Video: Video Editing using Image Diffusioncited by 318
  30. 2023Ablating Concepts in Text-to-Image Diffusion Modelscited by 312
  31. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 312
  32. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generationcited by 300
  33. 2023Text-to-3D using Gaussian Splattingcited by 299
  34. 2023Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Modelscited by 297
  35. 2024Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understandingcited by 294
  36. 2023The Stable Signature: Rooting Watermarks in Latent Diffusion Modelscited by 289
  37. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 284
  38. 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusioncited by 280

50 most cited of 921 All 921 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Reusing Computation in Text-to-Image Diffusion for Efficient Generation of Image Sets
  2. 2026ImageRAGTurbo: Towards One-step Text-to-Image Generation with Retrieval-Augmented Diffusion Models
  3. 2026Efficient Text-to-Image Generation: An Adaptive Step Schedule Controller for Diffusion Models
  4. 2026i1: A Simple and Fully Open Recipe for Strong Text-to-Image Models
  5. 2025Cost-Aware Routing for Efficient Text-To-Image Generation
  6. 2026Improving Controllable Generation: Faster Training and Better Performance via x0-Supervision
  7. 2024Enhancing Semantic Fidelity in Text-to-Image Synthesis: Attention Regulation in Diffusion Models
  8. 2022Implementing and Experimenting with Diffusion Models for Text-to-Image Generation