mlsanityData to 7 Oct 2026

ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment

arXiv 2403.05135 ↗First posted 8 Mar 2024cs.CVcited by 455

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Diffusion models have demonstrated remarkable performance in the domain of text-to-image generation. However, most widely used models still employ CLIP as their text encoder, which constrains their ability to comprehend dense prompts, encompassing multiple objects, detailed attributes, complex relationships, long-text alignment, etc. In this paper, we introduce an Efficient Large Language Model Adapter, termed ELLA, which equips text-to-image diffusion models with powerful Large Language Models (LLM) to enhance text alignment without training of either U-Net or LLM. To seamlessly bridge two pre-trained models, we investigate a range of semantic alignment connector designs and propose a novel module, the Timestep-Aware Semantic Connector (TSC), which dynamically extracts timestep-dependent conditions from LLM. Our approach adapts semantic features at different stages of the denoising process, assisting diffusion models in interpreting lengthy and intricate prompts over sampling timesteps. Additionally, ELLA can be readily incorporated with community models and tools to improve their prompt-following capabilities. To assess text-to-image models in dense prompt following, we introduce Dense Prompt Graph Benchmark (DPG-Bench), a challenging benchmark consisting of 1K dense prompts. Extensive experiments demonstrate the superiority of ELLA in dense prompt following compared to state-of-the-art methods, particularly in multiple object compositions involving diverse attributes and relationships.

Key facts. Cited by 455 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 51 of its 60 references that are in the map. First posted on arXiv 8 Mar 2024.

Builds on

51 of 60 references.

Closest ancestors · 19

Same direction, newest first

  1. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs22 Jan
  2. 2023Ranni: Taming Text-to-Image Diffusion for Accurate Instruction Following28 Nov
  3. 2023Paragraph-to-Image Generation with Information-Enriched Diffusion Model24 Nov
  4. 2023Dense Text-to-Image Generation with Attention Modulation24 Aug
  5. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models13 Aug
  6. 2023Divide & Bind Your Attention for Improved Generative Semantic Nursing20 Jul
  7. 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion20 Jul
  8. 2023Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignment15 Jun
  9. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models23 May
  10. 2023SUR-adapter: Enhancing Text-to-Image Pre-trained Diffusion Models with Large Language Models9 May
  11. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation16 Feb
  12. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
7 more
  1. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models31 Jan
  2. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis9 Dec
  3. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
  4. 2022Compositional Visual Generation with Composable Diffusion Models3 Jun
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  7. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2022Scalable Diffusion Models with Transformerscited by 5,931
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
20 more
  1. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  2. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  3. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  4. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  5. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  6. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 722
  7. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  8. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  9. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  10. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 312
  11. 2022Perception Prioritized Training of Diffusion Modelscited by 301
  12. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 278
  13. 2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 253
  14. 2022Optimizing Prompts for Text-to-Image Generationcited by 204
  15. 2023JourneyDB: A Benchmark for Generative Image Understandingcited by 203
  16. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
  17. 2023Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generationcited by 159
  18. 2023DiffiT: Diffusion Vision Transformers for Image Generationcited by 104
  19. 2023DreamSync: Aligning Text-to-Image Generation with Image Understanding Feedbackcited by 57
  20. 2023Visual Programming for Text-to-Image Generation and Evaluationcited by 53

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 40; 2025: 204; 2026: 211.

455 papers.

Sort:

By area:

  1. 2025Qwen-Image Technical Reportcited by 973
  2. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  3. 2024SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformerscited by 331
  4. 2024Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesiscited by 297
  5. 2025Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformercited by 247
  6. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
  7. 2025Transfer between Modalities with MetaQueriescited by 231
  8. 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 200
  9. 2025WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generationcited by 196
  10. 2025SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformercited by 165
  11. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 162
  12. 2024Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretrainingcited by 148
38 more
  1. 2025HiDream-I1: A High-Efficient Image Generative Foundation Model with Sparse Diffusion Transformercited by 131
  2. 2024HART: Efficient Visual Generation with Hybrid Autoregressive Transformercited by 128
  3. 2024ID-Animator: Zero-Shot Identity-Preserving Human Video Generationcited by 122
  4. 2025Emu3.5: Native Multimodal Models are World Learnerscited by 119
  5. 2024Playground v3: Improving Text-to-Image Alignment with Deep-Fusion Large Language Modelscited by 113
  6. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  7. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 105
  8. 2025Echo-4o: Harnessing the Power of GPT-4o Synthetic Images for Improved Image Generationcited by 95
  9. 2025ShareGPT-4o-Image: Aligning Multimodal Models with GPT-4o-Level Image Generationcited by 94
  10. 2025Lumina-Image 2.0: A Unified and Efficient Image Generative Frameworkcited by 93
  11. 2025LongCat-Image Technical Reportcited by 92
  12. 2025PixelFlow: Pixel-Space Generative Models with Flowcited by 91
  13. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  14. 2025SimpleAR: Pushing the Frontier of Autoregressive Visual Generation through Pretraining, SFT, and RLcited by 85
  15. 2025Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understandingcited by 83
  16. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  17. 2025OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generationcited by 81
  18. 2025PixNerd: Pixel Neural Field Diffusioncited by 75
  19. 2025PixelDiT: Pixel Diffusion Transformers for Image Generationcited by 75
  20. 2025Thinking with Generated Imagescited by 71
  21. 2025Goku: Flow Based Video Generative Foundation Modelscited by 67
  22. 2025Ovis-U1 Technical Reportcited by 65
  23. 2025TIIF-Bench: How Does Your T2I Model Follow Your Instructions?cited by 64
  24. 2025Ming-Omni: A Unified Multimodal Model for Perception and Generationcited by 62
  25. 2024CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matchingcited by 60
  26. 2025XVerse: Consistent Multi-Subject Control of Identity and Semantic Attributes via DiT Modulationcited by 55
  27. 2025NextStep-1: Toward Autoregressive Image Generation with Continuous Tokens at Scalecited by 55
  28. 2025DeCo: Frequency-Decoupled Pixel Diffusion for End-to-End Image Generationcited by 55
  29. 2024Faster Diffusion via Temporal Attention Decompositioncited by 54
  30. 2025OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generationcited by 50
  31. 2025BLIP3o-NEXT: Next Frontier of Native Image Generationcited by 50
  32. 2026Scaling Text-to-Image Diffusion Transformers with Representation Autoencoderscited by 49
  33. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 46
  34. 2025FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocitiescited by 46
  35. 2024Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Modelscited by 45
  36. 2025OpenUni: A Simple Baseline for Unified Multimodal Understanding and Generationcited by 45
  37. 2025InfinityStar: Unified Spacetime AutoRegressive Modeling for Visual Generationcited by 43
  38. 2025Distribution Matching Distillation Meets Reinforcement Learningcited by 43

50 most cited of 455 All 455 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024AlignIT: Enhancing Prompt Alignment in Customization of Text-to-Image Models
  2. 2024Bridging the Gap: Aligning Text-to-Image Diffusion Models with Specific Feedback
  3. 2025Decoder-Only LLMs are Better Controllers for Diffusion Models
  4. 2024Noise Diffusion for Enhancing Semantic Faithfulness in Text-to-Image Synthesis
  5. 2024Information Theoretic Text-to-Image Alignment
  6. 2025M3T2IBench: A Large-Scale Multi-Category, Multi-Instance, Multi-Relation Text-to-Image Benchmark
  7. 2024Towards Better Text-to-Image Generation Alignment via Attention Modulation
  8. 2024FRAP: Faithful and Realistic Text-to-Image Generation with Adaptive Prompt Weighting