mlsanityData to 7 Oct 2026

Ranni: Taming Text-to-Image Diffusion for Accurate Instruction Following

arXiv 2311.17002 ↗First posted 28 Nov 2023cs.CVcited by 77

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Existing text-to-image (T2I) diffusion models usually struggle in interpreting complex prompts, especially those with quantity, object-attribute binding, and multi-subject descriptions. In this work, we introduce a semantic panel as the middleware in decoding texts to images, supporting the generator to better follow instructions. The panel is obtained through arranging the visual concepts parsed from the input text by the aid of large language models, and then injected into the denoising network as a detailed control signal to complement the text condition. To facilitate text-to-panel learning, we come up with a carefully designed semantic formatting protocol, accompanied by a fully-automatic data preparation pipeline. Thanks to such a design, our approach, which we call Ranni, manages to enhance a pre-trained T2I generator regarding its textual controllability. More importantly, the introduction of the generative middleware brings a more convenient form of interaction (i.e., directly adjusting the elements in the panel or using language instructions) and further allows users to finely customize their generation, based on which we develop a practical system and showcase its potential in continuous generation and chatting-based editing. Our project page is at https://ranni-t2i.github.io/Ranni.

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 48 of its 54 references that are in the map. First posted on arXiv 28 Nov 2023.

Builds on

48 of 54 references.

Closest ancestors · 16

Same direction, newest first

  1. 2023Check, Locate, Rectify: A Training-Free Layout Calibration System for Text-to-Image Generation27 Nov
  2. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models23 May
  3. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models16 Feb
  4. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  5. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models31 Jan
  6. 2023GLIGEN: Open-Set Grounded Text-to-Image Generation17 Jan
  7. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis9 Dec
  8. 2022Sketch-Guided Text-to-Image Diffusion Models24 Nov
  9. 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov
  10. 2022DiffEdit: Diffusion-based semantic image editing with mask guidance20 Oct
  11. 2022Compositional Visual Generation with Composable Diffusion Models3 Jun
  12. 2022Pretraining is All You Need for Image-to-Image Translation25 May
4 more
  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  3. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  4. 2021Blended Diffusion for Text-driven Editing of Natural Images29 Nov

Other references · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2020Denoising Diffusion Implicit Modelscited by 9,655
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  10. 2023Segment Anythingcited by 8,158
  11. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  12. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
20 more
  1. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  3. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  4. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  5. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  6. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  7. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  8. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  9. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  10. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 615
  11. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  12. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  13. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  14. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 303
  15. 2022SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Modelcited by 276
  16. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  17. 2017WebVision Database: Visual Learning and Understanding from Web Datacited by 262
  18. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  19. 2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 253
  20. 2023Visual Programming for Text-to-Image Generation and Evaluationcited by 53

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 28; 2025: 38; 2026: 11.

77 papers.

Sort:

By area:

  1. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  2. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 455
  3. 2024CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Image Generationcited by 58
  4. 2024VAR-CLIP: Text-to-Image Generator with Visual Auto-Regressive Modelingcited by 42
  5. 2025Reflect-DiT: Inference-Time Scaling for Text-to-Image Diffusion Transformers via In-Context Reflectioncited by 42
  6. 2024Compositional Text-to-Image Generation with Dense Blob Representationscited by 38
  7. 2024Region-Aware Text-to-Image Generation via Hard Binding and Soft Refinementcited by 37
  8. 20243DIS: Depth-Driven Decoupled Instance Synthesis for Text-to-Image Generationcited by 36
  9. 2025CreatiDesign: A Unified Multi-Conditional Diffusion Transformer for Creative Graphic Designcited by 36
  10. 2024Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesiscited by 35
  11. 2024How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization?cited by 32
  12. 2024All Seeds Are Not Equal: Enhancing Compositional Text-to-Image Generation with Reliable Random Seedscited by 31
38 more
  1. 2024Conditional Image Synthesis with Diffusion Models: A Surveycited by 29
  2. 2025DreamRenderer: Taming Multi-Instance Attribute Control in Large-Scale Text-to-Image Modelscited by 28
  3. 2025PhotoDoodle: Learning Artistic Image Editing from Few-Shot Pairwise Datacited by 23
  4. 2024ResMaster: Mastering High-Resolution Image Generation via Structural and Fine-Grained Guidancecited by 21
  5. 2024Magnet: We Never Know How Text-to-Image Diffusion Models Work, Until We Learn How Vision-Language Models Functioncited by 19
  6. 2024TheaterGen: Character Management with LLM for Consistent Multi-turn Image Generationcited by 18
  7. 2024Improving Long-Text Alignment for Text-to-Image Diffusion Modelscited by 18
  8. 2025PlanGen: Towards Unified Layout Planning and Image Generation in Auto-Regressive Vision Language Modelscited by 18
  9. 2024Mimir: Improving Video Diffusion Models for Precise Text Understandingcited by 16
  10. 2024HiPrompt: Tuning-free Higher-Resolution Generation with Hierarchical MLLM Promptscited by 14
  11. 2025Towards Transformer-Based Aligned Generation with Self-Coherence Guidancecited by 14
  12. 2025Ming-Lite-Uni: Advancements in Unified Architecture for Natural Multimodal Interactioncited by 13
  13. 2024ChatGen: Automatic Text-to-Image Generation From FreeStyle Chattingcited by 11
  14. 2024MuLan: Multimodal-LLM Agent for Progressive and Interactive Multi-Object Diffusioncited by 10
  15. 2025Be Decisive: Noise-Induced Layouts for Multi-Subject Generationcited by 10
  16. 2025Seg2Any: Open-set Segmentation-Mask-to-Image Generation with Precise Shape and Semantic Controlcited by 10
  17. 2025YOLO-Count: Differentiable Object Counting for Text-to-Image Generationcited by 10
  18. 2024Story3D-Agent: Exploring 3D Storytelling Visualization with Large Language Modelscited by 9
  19. 2024ROICtrl: Boosting Instance Control for Visual Generationcited by 9
  20. 2025InstanceAssemble: Layout-Aware Image Generation via Instance Assembling Attentioncited by 9
  21. 2025OverLayBench: A Benchmark for Layout-to-Image Generation with Dense Overlapscited by 9
  22. 2025MammothModa2: A Unified AR-Diffusion Framework for Multimodal Understanding and Generationcited by 9
  23. 2025SynMotion: Semantic-Visual Adaptation for Motion Customized Video Generationcited by 7
  24. 2025LayerEdit: Disentangled Multi-Object Editing via Conflict-Aware Multi-Layer Learningcited by 7
  25. 2024Draw Like an Artist: Complex Scene Generation with Diffusion Model via Composition, Painting, and Retouchingcited by 6
  26. 2024KnobGen: Controlling the Sophistication of Artwork in Sketch-Based Diffusion Modelscited by 6
  27. 2025Multi-party Collaborative Attention Control for Image Customizationcited by 6
  28. 2025Lay-Your-Scene: Natural Scene Layout Generation with Diffusion Transformerscited by 6
  29. 2025Dimension-Reduction Attack! Video Generative Models are Experts on Controllable Image Synthesiscited by 6
  30. 2025Inversion-DPO: Precise and Efficient Post-Training for Diffusion Modelscited by 6
  31. 2025SUB: Benchmarking CBM Generalization via Synthetic Attribute Substitutionscited by 6
  32. 2025The Promise of RL for Autoregressive Image Editingcited by 6
  33. 2025IdentityStory: Taming Your Identity-Preserving Generator for Human-Centric Story Generationcited by 6
  34. 2025Geometrical Properties of Text Token Embeddings for Strong Semantic Binding in Text-to-Image Generationcited by 5
  35. 2024DA-HFNet: Progressive Fine-Grained Forgery Image Detection and Localization Based on Dual Attentioncited by 4
  36. 2024Imagining the Unseen: Generative Location Modeling for Object Placementcited by 4
  37. 2024GraPE: A Generate-Plan-Edit Framework for Compositional T2I Synthesiscited by 4
  38. 2025POEM: Precise Object-level Editing via MLLM controlcited by 4

50 most cited of 77 All 77 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Decoder-Only LLMs are Better Controllers for Diffusion Models
  2. 2025Few-Step Distillation for Text-to-Image Generation: A Practical Guide
  3. 2025DeCoT: Decomposing Complex Instructions for Enhanced Text-to-Image Generation with Large Language Models
  4. 2024Controllable Generation with Text-to-Image Diffusion Models: A Survey
  5. 2025T2I-Copilot: A Training-Free Multi-Agent Text-to-Image System for Enhanced Prompt Interpretation and Interactive Generation
  6. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation
  7. 2023Compositional Text-to-Image Synthesis with Attention Map Control of Diffusion Models
  8. 2023Paragraph-to-Image Generation with Information-Enriched Diffusion Model