mlsanityData to 14 Oct 2026

Towards Understanding the Working Mechanism of Text-to-Image Diffusion Model

arXiv 2405.15330 ↗First posted 24 May 2024cs.CV cs.LGcited by 48

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Recently, the strong latent Diffusion Probabilistic Model (DPM) has been applied to high-quality Text-to-Image (T2I) generation (e.g., Stable Diffusion), by injecting the encoded target text prompt into the gradually denoised diffusion image generator. Despite the success of DPM in practice, the mechanism behind it remains to be explored. To fill this blank, we begin by examining the intermediate statuses during the gradual denoising generation process in DPM. The empirical observations indicate, the shape of image is reconstructed after the first few denoising steps, and then the image is filled with details (e.g., texture). The phenomenon is because the low-frequency signal (shape relevant) of the noisy image is not corrupted until the final stage in the forward process (initial stage of generation) of adding noise in DPM. Inspired by the observations, we proceed to explore the influence of each token in the text prompt during the two stages. After a series of experiments of T2I generations conditioned on a set of text prompts. We conclude that in the earlier generation stage, the image is mostly decided by the special token [textttEOS] in the text prompt, and the information in the text prompt is already conveyed in this stage. After that, the diffusion model completes the details of generated images by information from themselves. Finally, we propose to apply this observation to accelerate the process of T2I generation by properly removing text guidance, which finally accelerates the sampling up to 25%+.

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 46 of its 53 references that are in the map. First posted on arXiv 24 May 2024.

Builds on

46 of 53 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Faster Diffusion via Temporal Attention Decomposition3 Apr
  2. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
  3. 2022What the DAAM: Interpreting Stable Diffusion Using Cross Attention10 Oct
  4. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr

Other references · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2023GPT-4 Technical Reportcited by 18,396
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  9. 2020Denoising Diffusion Implicit Modelscited by 9,781
  10. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  12. 2022Scalable Diffusion Models with Transformerscited by 6,054
28 more
  1. 2022Classifier-Free Diffusion Guidancecited by 5,847
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  4. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  5. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  7. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  9. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
  10. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  11. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,290
  12. 2022Diffusion Models in Vision: A Surveycited by 1,015
  13. 2021Image Segmentation Using Text and Image Promptscited by 512
  14. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 502
  15. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  16. 2023AnyDoor: Zero-shot Object-level Image Customizationcited by 412
  17. 2023PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embeddingcited by 328
  18. 2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 260
  19. 2023FreeU: Free Lunch in Diffusion U-Netcited by 196
  20. 2023Dreamix: Video Diffusion Models are General Video Editorscited by 191
  21. 2023StableVideo: Text-driven Consistency-aware Diffusion Video Editingcited by 183
  22. 2021Counterfactual Memorization in Neural Language Modelscited by 156
  23. 2022Diffusion Probabilistic Model Made Slimcited by 121
  24. 2021Do Long-Range Language Models Actually Use Long-Range Context?cited by 92
  25. 2020Generate Your Counterfactuals: Towards Controlled Counterfactual Generation for Textcited by 75
  26. 2022RankGen: Improving Text Generation with Large Ranking Modelscited by 72
  27. 2023Latent Diffusion Counterfactual Explanationscited by 17
  28. 2021OodGAN: Generative Adversarial Network for Out-of-Domain Data Generationcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 28; 2026: 17.

48 papers.

Sort:

By area:

  1. 2025Lumina-Image 2.0: A Unified and Efficient Image Generative Frameworkcited by 96
  2. 2025Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understandingcited by 88
  3. 2025A Fourier Space Perspective on Diffusion Modelscited by 52
  4. 2025Lumina-mGPT 2.0: Stand-Alone AutoRegressive Image Modelingcited by 37
  5. 2025HUD: Hierarchical Uncertainty-Aware Disambiguation Network for Composed Video Retrievalcited by 32
  6. 2025SuperEdit: Rectifying and Facilitating Supervision for Instruction-Based Image Editingcited by 18
  7. 2025UltraHR-100K: Enhancing UHR Image Synthesis with A Large-Scale High-Quality Datasetcited by 18
  8. 2025Enhancing Intent Understanding for Ambiguous prompt: A Human-Machine Co-Adaption Strategycited by 16
  9. 2025TIDE : Temporal-Aware Sparse Autoencoders for Interpretable Diffusion Transformers in Image Generationcited by 16
  10. 2026LUVE : Latent-Cascaded Ultra-High-Resolution Video Generation with Dual Frequency Expertscited by 15
  11. 2024Text Embedding is Not All You Need: Attention Control for Text-to-Image Semantic Alignment with Text Self-Attention Mapscited by 12
  12. 2025ForgetMe: Evaluating Selective Forgetting in Generative Modelscited by 11
36 more
  1. 2025Twin Co-Adaptive Dialogue for Progressive Image Generationcited by 11
  2. 2025Resurrect Mask AutoRegressive Modeling for Efficient and Scalable Image Generationcited by 9
  3. 2024TIPO: Text to Image with Text Presampling for Prompt Optimizationcited by 6
  4. 2025RLGF: Reinforcement Learning with Geometric Feedback for Autonomous Driving Video Generationcited by 5
  5. 2025Mitigating Semantic Collapse in Partially Relevant Video Retrievalcited by 5
  6. 2024FreeCond: Free Lunch in the Input Conditions of Text-Guided Inpaintingcited by 4
  7. 2025What Lurks Within? Concept Auditing for Shared Diffusion Models at Scalecited by 4
  8. 2025Deeper Diffusion Models Amplify Biascited by 4
  9. 2025Improved Diffusion-based Generative Model with Better Adversarial Robustnesscited by 3
  10. 2025Exploring Position Encoding in Diffusion U-Net for Training-free High-resolution Image Generationcited by 3
  11. 2025Dynamic Attention Analysis for Backdoor Detection in Text-to-Image Diffusion Modelscited by 3
  12. 2026TexTailor: Inference-Time Textual Guidance Tailoring for Multimodal Diffusion Transformerscited by 3
  13. 2026Elucidating the SNR-t Bias of Diffusion Probabilistic Modelscited by 3
  14. 2025ResDiT: Evoking the Intrinsic Resolution Scalability in Diffusion Transformerscited by 2
  15. 2025dMLLM-TTS: Self-Verified and Efficient Test-Time Scaling for Diffusion Multi-Modal Large Language Modelscited by 2
  16. 2026Realiz3D: 3D Generation Made Photorealistic via Domain-Aware Learningcited by 2
  17. 2026Registers Matter for Pixel-Space Diffusion Transformerscited by 2
  18. 2026Why Do Video Diffusion Models Violate Physics? Unveiling the Flaws in Attention Mechanismscited by 2
  19. 2025TDRI: Two-Phase Dialogue Refinement and Co-Adaptation for Interactive Image Generationcited by 1
  20. 2025Enhancing Membership Inference Attacks on Diffusion Models from a Frequency-Domain Perspectivecited by 1
  21. 2025CPO: Condition Preference Optimization for Controllable Image Generationcited by 1
  22. 2026Fragile Reconstruction: Adversarial Vulnerability of Reconstruction-Based Detectors for Diffusion-Generated Imagescited by 1
  23. 2026Explore or Converge? Stage-Guided Per-Step Optimization for Diffusion Modelscited by 1
  24. 2025Multi-Modal Machine Learning Framework for Predicting Early Recurrence of Brain Tumors Using MRI and Clinical Biomarkerscited by 0
  25. 2025A Multimodal Deep Learning Framework for Early Diagnosis of Liver Cancer via Optimized BiLSTM-AM-VMD Architecturecited by 0
  26. 2025SP-Guard: Selective Prompt-adaptive Guidance for Safe Text-to-Image Generationcited by 0
  27. 2025Food Image Generation on Multi-Noun Categoriescited by 0
  28. 2026Training-Free Text-to-Image Compositional Food Generation via Prompt Graftingcited by 0
  29. 2026ADAPT: Attention Driven Adaptive Prompt Scheduling and InTerpolating Orthogonal Complements for Rare Concepts Generationcited by 0
  30. 2026Attention Frequency Modulation: Training-Free Spectral Modulation of Diffusion Cross-Attentioncited by 0
  31. 2026OFA-Diffusion Compression: Compressing Diffusion Model in One-Shot Mannercited by 0
  32. 2026Memorization In Stable Diffusion Is Unexpectedly Driven by CLIP Embeddingscited by 0
  33. 2026CLIP Tricks You: Training-free Token Pruning for Efficient Pixel Grounding in Large VIsion-Language Modelscited by 0
  34. 2026FairFlow: Demystifying and Mitigating Stereotype Bias in Text-to-Image Diffusion Transformerscited by 0
  35. 2026DiffReID: Discriminative Diffusion Model for Object Re-Identificationcited by 0
  36. 2026Rethinking Memorization Mitigation in Diffusion Models: Reinforcing Text Conditioningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Controllable Generation with Text-to-Image Diffusion Models: A Survey
  2. 2024RDPM: Solve Diffusion Probabilistic Models via Recurrent Token Prediction
  3. 2026Toward Early Quality Assessment of Text-to-Image Diffusion Models
  4. 2021Structured Denoising Diffusion Models in Discrete State-Spaces
  5. 2023Controlled and Conditional Text to Image Generation with Diffusion Prior
  6. 2024Step Saver: Predicting Minimum Denoising Steps for Diffusion Model Image Generation
  7. 2026ImageRAGTurbo: Towards One-step Text-to-Image Generation with Retrieval-Augmented Diffusion Models
  8. 2024Diffusion Model Compression for Image-to-Image Translation