mlsanityData to 14 Oct 2026

ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Experts

arXiv 2210.15257 ↗First posted 27 Oct 2022cs.CV cs.AIcited by 112

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Recent progress in diffusion models has revolutionized the popular technology of text-to-image generation. While existing approaches could produce photorealistic high-resolution images with text conditions, there are still several open problems to be solved, which limits the further improvement of image fidelity and text relevancy. In this paper, we propose ERNIE-ViLG 2.0, a large-scale Chinese text-to-image diffusion model, to progressively upgrade the quality of generated images by: (1) incorporating fine-grained textual and visual knowledge of key elements in the scene, and (2) utilizing different denoising experts at different denoising stages. With the proposed mechanisms, ERNIE-ViLG 2.0 not only achieves a new state-of-the-art on MS-COCO with zero-shot FID score of 6.75, but also significantly outperforms recent models in terms of image fidelity and image-text alignment, with side-by-side human evaluation on the bilingual prompt set ViLG-300.

Key facts. Cited by 112 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 44 of its 49 references that are in the map. First posted on arXiv 27 Oct 2022.

Builds on

44 of 49 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  3. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  8. 2022Classifier-Free Diffusion Guidancecited by 5,847
  9. 2022Generative Adversarial Networkscited by 5,627
  10. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  12. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
29 more
  1. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  2. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  3. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  4. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  5. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 834
  6. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  7. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  8. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  9. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  10. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  11. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
  12. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  13. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 307
  14. 2019ERNIE: Enhanced Representation through Knowledge Integrationcited by 302
  15. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 260
  16. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  17. 2021Hash Layers For Large Sparse Modelscited by 237
  18. 2019ERNIE 2.0: A Continual Pre-training Framework for Language Understandingcited by 231
  19. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  20. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 203
  21. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 191
  22. 2020DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesiscited by 162
  23. 2022A Review of Sparse Expert Models in Deep Learningcited by 148
  24. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 132
  25. 2021LAFITE: Towards Language-Free Training for Text-to-Image Generationcited by 128
  26. 2022A very preliminary analysis of DALL-E 2cited by 82
  27. 2022ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understandingcited by 67
  28. 2020Scalable Transfer Learning with Expert Modelscited by 49
  29. 2021ERNIE-ViLG: Unified Generative Pre-training for Bidirectional Vision-Language Generationcited by 41

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 47; 2024: 31; 2025: 16; 2026: 14.

112 papers.

Sort:

By area:

  1. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 864
  2. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 699
  3. 2023Simple diffusion: End-to-end diffusion for high resolution imagescited by 422
  4. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 385
  5. 2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preferencecited by 287
  6. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 265
  7. 2023Efficient Diffusion Training via Min-SNR Weighting Strategycited by 246
  8. 2023TextDiffuser: Diffusion Models as Text Painterscited by 229
  9. 2023Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuningcited by 191
  10. 2023Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Modelscited by 169
  11. 2023Dense Text-to-Image Generation with Attention Modulationcited by 163
  12. 2023Holistic Evaluation of Text-To-Image Modelscited by 160
38 more
  1. 2023Cache Me if You Can: Accelerating Diffusion Models through Block Cachingcited by 151
  2. 2022NeuralLift-360: Lifting An In-the-wild 2D Photo to A 3D Object with 360deg Viewscited by 148
  3. 2023RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Pathscited by 142
  4. 2023SimDA: Simple Diffusion Adapter for Efficient Video Generationcited by 101
  5. 2024Scaling Diffusion Transformers to 16 Billion Parameterscited by 66
  6. 2024CRS-Diff: Controllable Remote Sensing Image Generation with Diffusion Modelcited by 55
  7. 2024QuEST: Low-bit Diffusion Model Quantization via Efficient Selective Finetuningcited by 49
  8. 2023RenAIssance: A Survey into AI Text-to-Image Generation in the Era of Large Modelcited by 47
  9. 2024Diffusion Model-Based Video Editing: A Surveycited by 44
  10. 2023TokenCompose: Text-to-Image Diffusion with Token-level Supervisioncited by 40
  11. 2023Paragraph-to-Image Generation with Information-Enriched Diffusion Modelcited by 39
  12. 2024Improved Noise Schedule for Diffusion Trainingcited by 39
  13. 2022ObjectStitch: Generative Object Compositingcited by 36
  14. 2022Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Modelcited by 36
  15. 2023GlyphDraw: Seamlessly Rendering Text with Intricate Spatial Structures in Text-to-Image Generationcited by 34
  16. 2024Switch Diffusion Transformer: Synergizing Denoising Tasks with Sparse Mixture-of-Expertscited by 31
  17. 2023AltDiffusion: A Multilingual Text-to-Image Diffusion Modelcited by 30
  18. 2024EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routingcited by 30
  19. 2024DreamOmni: Unified Image Generation and Editingcited by 30
  20. 2025Routing Matters in MoE: Scaling Diffusion Transformers with Explicit Routing Guidancecited by 26
  21. 2023A Survey of Diffusion Based Image Generation Models: Issues and Their Solutionscited by 25
  22. 2023LLMGA: Multimodal Large Language Model based Generation Assistantcited by 25
  23. 2025DiffMoE: Dynamic Token Selection for Scalable Diffusion Transformerscited by 24
  24. 2025Phased DMD: Few-step Distribution Matching Distillation via Score Matching within Subintervalscited by 24
  25. 2023Modulating Pretrained Diffusion Models for Multimodal Image Synthesiscited by 23
  26. 2023Improved Visual Story Generation with Adaptive Context Modelingcited by 22
  27. 2023Mini-DALLE3: Interactive Text to Image by Prompting Large Language Modelscited by 22
  28. 2024SCoFT: Self-Contrastive Fine-Tuning for Equitable Image Generationcited by 19
  29. 2024An Upload-Efficient Scheme for Transferring Knowledge From a Server-Side Pre-trained Generator to Clients in Heterogeneous Federated Learningcited by 17
  30. 2024SimpleSpeech 2: Towards Simple and Efficient Text-to-Speech with Flow-based Scalar Latent Transformer Diffusion Modelscited by 17
  31. 2023From Text to Mask: Localizing Entities Using the Attention of Text-to-Image Diffusion Modelscited by 16
  32. 2023TP2O: Creative Text Pair-to-Object Generation using Balance Swap-Samplingcited by 16
  33. 2023Nonparametric Generative Modeling with Conditional Sliced-Wasserstein Flowscited by 15
  34. 2023MultiFusion: Fusing Pre-Trained Models for Multi-Lingual, Multi-Modal Image Generationcited by 15
  35. 2024Bora: Biomedical Generalist Video Generation Modelcited by 15
  36. 2024MoLE: Enhancing Human-centric Text-to-image Diffusion via Mixture of Low-rank Expertscited by 15
  37. 2023MAGVLT: Masked Generative Vision-and-Language Transformercited by 14
  38. 2025Expert Race: A Flexible Routing Strategy for Scaling Diffusion Transformer with Mixture of Expertscited by 14

50 most cited of 112 All 112 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026TEXTS-Diff: TEXTS-Aware Diffusion Model for Real-World Text Image Super-Resolution
  2. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers
  3. 2024Taiyi-Diffusion-XL: Advancing Bilingual Text-to-Image Generation with Large Vision-Language Model Support
  4. 2024Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding
  5. 2025Text-Aware Image Restoration with Diffusion Models
  6. 2025Multimodal LLM-Guided Semantic Correction in Text-to-Image Diffusion
  7. 2025Development and Enhancement of Text-to-Image Diffusion Models
  8. 2023Diffusion-based Blind Text Image Super-Resolution