mlsanityData to 7 Oct 2026

RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Paths

arXiv 2305.18295 ↗First posted 29 May 2023cs.CVcited by 141

On the map: Diffusion models → Image generation, text-to-image and editing

also in 1 more
Abstract (arXiv)

Text-to-image generation has recently witnessed remarkable achievements. We introduce a text-conditional image diffusion model, termed RAPHAEL, to generate highly artistic images, which accurately portray the text prompts, encompassing multiple nouns, adjectives, and verbs. This is achieved by stacking tens of mixture-of-experts (MoEs) layers, i.e., space-MoE and time-MoE layers, enabling billions of diffusion paths (routes) from the network input to the output. Each path intuitively functions as a "painter" for depicting a particular textual concept onto a specified image region at a diffusion timestep. Comprehensive experiments reveal that RAPHAEL outperforms recent cutting-edge models, such as Stable Diffusion, ERNIE-ViLG 2.0, DeepFloyd, and DALL-E 2, in terms of both image quality and aesthetic appeal. Firstly, RAPHAEL exhibits superior performance in switching images across diverse styles, such as Japanese comics, realism, cyberpunk, and ink illustration. Secondly, a single model with three billion parameters, trained on 1,000 A100 GPUs for two months, achieves a state-of-the-art zero-shot FID score of 6.61 on the COCO dataset. Furthermore, RAPHAEL significantly surpasses its counterparts in human evaluation on the ViLG-300 benchmark. We believe that RAPHAEL holds the potential to propel the frontiers of image generation research in both academia and industry, paving the way for future breakthroughs in this rapidly evolving field. More details can be found on a webpage: https://raphael-painter.github.io/.

Key facts. Cited by 141 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 54 of its 65 references that are in the map. First posted on arXiv 29 May 2023.

Builds on

54 of 65 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models25 May
  2. 2023Cones: Concept Neurons in Diffusion Models for Customized Generation9 Mar
  3. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  4. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
  5. 2022ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Experts27 Oct
  6. 2022Imagic: Text-Based Real Image Editing with Diffusion Models17 Oct
  7. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  10. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2022Training language models to follow instructions with human feedbackcited by 15,712
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  9. 2020Denoising Diffusion Implicit Modelscited by 9,655
  10. 2022Classifier-Free Diffusion Guidancecited by 5,717
  11. 2022Generative Adversarial Networkscited by 5,603
  12. 2017Neural Discrete Representation Learningcited by 4,650
32 more
  1. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  3. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  7. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  8. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  10. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  11. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  12. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  13. 2019An Introduction to Variational Autoencoderscited by 997
  14. 2021Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Datacited by 961
  15. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  16. 2017Focal Loss for Dense Object Detectioncited by 749
  17. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  18. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  19. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  20. 2015Holistically-Nested Edge Detectioncited by 590
  21. 2017Generative Adversarial Networks: An Overviewcited by 588
  22. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  23. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  24. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  25. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  26. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  27. 2023One Transformer Fits All Distributions in Multi-Modal Diffusion at Scalecited by 185
  28. 2020DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesiscited by 161
  29. 2023Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generationcited by 128
  30. 2021FastMoE: A Fast Mixture-of-Expert Training Systemcited by 110
  31. 2023Scaling Vision-Language Models with Sparse Mixture of Expertscited by 88
  32. 2022Large-batch Optimization for Dense Visual Predictionscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 30; 2024: 64; 2025: 30; 2026: 17.

141 papers.

Sort:

By area:

  1. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  2. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  3. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  4. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  5. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  6. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  7. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  8. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  9. 2024Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wildcited by 281
  10. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
  11. 2023Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Modelscited by 169
  12. 2023UFOGen: You Forward Once Large Scale Text-to-Image Generation via Diffusion GANscited by 165
38 more
  1. 2024DragAnything: Motion Control for Anything using Entity Representationcited by 159
  2. 2023Cache Me if You Can: Accelerating Diffusion Models through Block Cachingcited by 149
  3. 2024Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokenscited by 147
  4. 2023SSR-Encoder: Encoding Selective Subject Representation for Subject-Driven Generationcited by 142
  5. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  6. 2024InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimizationcited by 123
  7. 2023Cones 2: Customizable Image Synthesis with Multiple Subjectscited by 101
  8. 2023SimDA: Simple Diffusion Adapter for Efficient Video Generationcited by 101
  9. 2023GenTron: Diffusion Transformers for Image and Video Generationcited by 80
  10. 2023PortraitBooth: A Versatile Portrait Model for Fast Identity-preserved Personalizationcited by 77
  11. 2023Matryoshka Diffusion Modelscited by 70
  12. 2024One-Step Diffusion Distillation through Score Implicit Matchingcited by 68
  13. 2023The Blessing of Randomness: SDE Beats ODE in General Diffusion-based Image Editingcited by 61
  14. 2023Focus on Your Instruction: Fine-grained and Multi-instruction Image Editing by Attention Modulationcited by 61
  15. 2024CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matchingcited by 60
  16. 2023On the Design Fundamentals of Diffusion Models: A Surveycited by 59
  17. 2023MomentDiff: Generative Video Moment Retrieval from Random to Realcited by 58
  18. 2024Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Modelscited by 45
  19. 2023LivePhoto: Real Image Animation with Text-guided Motion Controlcited by 44
  20. 2023Boosting Latent Diffusion with Flow Matchingcited by 44
  21. 2024ConsistentID: Portrait Generation with Multimodal Fine-Grained Identity Preservingcited by 42
  22. 2024SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Foldingcited by 42
  23. 2023TokenCompose: Text-to-Image Diffusion with Token-level Supervisioncited by 40
  24. 2023Paragraph-to-Image Generation with Information-Enriched Diffusion Modelcited by 39
  25. 2024Autoregressive Models in Vision: A Surveycited by 39
  26. 2024Alleviating Distortion in Image Generation via Multi-Resolution Diffusion Models and Time-Dependent Layer Normalizationcited by 36
  27. 2024Anim-Director: A Large Multimodal Model Powered Agent for Controllable Animation Video Generationcited by 35
  28. 2024Switch Diffusion Transformer: Synergizing Denoising Tasks with Sparse Mixture-of-Expertscited by 31
  29. 2024Erasing Concepts from Text-to-Image Diffusion Models with Few-shot Unlearningcited by 29
  30. 2024EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routingcited by 29
  31. 2024Unifying Bayesian Flow Networks and Diffusion Models through Stochastic Differential Equationscited by 28
  32. 2024TextCraftor: Your Text Encoder Can be Image Quality Controllercited by 27
  33. 2024Flowing from Words to Pixels: A Noise-Free Framework for Cross-Modality Evolutioncited by 26
  34. 2023LLMGA: Multimodal Large Language Model based Generation Assistantcited by 25
  35. 2024You Only Sample Once: Taming One-Step Text-to-Image Synthesis by Self-Cooperative Diffusion GANscited by 25
  36. 2024Diffusion Soup: Model Merging for Text-to-Image Diffusion Modelscited by 25
  37. 2024Next Patch Prediction for Autoregressive Visual Generationcited by 25
  38. 2024SmartControl: Enhancing ControlNet for Handling Rough Visual Conditionscited by 24

50 most cited of 141 All 141 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Text-to-image Diffusion Models in Generative AI: A Survey
  2. 2024ARTIST: Improving the Generation of Text-rich Images with Disentangled Diffusion Models and Large Language Models
  3. 2023SEGA: Instructing Text-to-Image Models using Semantic Guidance
  4. 2024UNIMO-G: Unified Image Generation through Multimodal Conditional Diffusion
  5. 2024Training-Free Sketch-Guided Diffusion with Latent Optimization
  6. 2024Image Regeneration: Evaluating Text-to-Image Model via Generating Identical Image with Multimodal Large Language Models
  7. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation
  8. 2024ORACLE: Leveraging Mutual Information for Consistent Character Generation with LoRAs in Diffusion Models