mlsanityData to 14 Oct 2026

Training-free Regional Prompting for Diffusion Transformers

arXiv 2411.02395 ↗First posted 4 Nov 2024cs.CVcited by 27

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Diffusion models have demonstrated excellent capabilities in text-to-image generation. Their semantic understanding (i.e., prompt following) ability has also been greatly improved with large language models (e.g., T5, Llama). However, existing models cannot perfectly handle long and complex text prompts, especially when the text prompts contain various objects with numerous attributes and interrelated spatial relationships. While many regional prompting methods have been proposed for UNet-based models (SD1.5, SDXL), but there are still no implementations based on the recent Diffusion Transformer (DiT) architecture, such as SD3 and FLUX.1.In this report, we propose and implement regional prompting for FLUX.1 based on attention manipulation, which enables DiT with fined-grained compositional text-to-image generation capability in a training-free manner. Code is available at https://github.com/antonioo-c/Regional-Prompting-FLUX.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 23 of its 27 references that are in the map. First posted on arXiv 4 Nov 2024.

Builds on

23 of 27 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024InstanceDiffusion: Instance-level Control for Image Generation5 Feb
  2. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs22 Jan
  3. 2023Dense Text-to-Image Generation with Attention Modulation24 Aug
  4. 2023Compositional Text-to-Image Synthesis with Attention Map Control of Diffusion Models23 May
  5. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation16 Feb
  6. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  7. 2023GLIGEN: Open-Set Grounded Text-to-Image Generation17 Jan

Other references · 16

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  7. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  8. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  9. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  10. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 286
  11. 2024MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidancecited by 144
  12. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
4 more
  1. 2024Playground v3: Improving Text-to-Image Alignment with Deep-Fusion Large Language Modelscited by 116
  2. 2023Mixture of Diffusers for scene composition and high resolution image generationcited by 56
  3. 2024PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructionscited by 31
  4. 2019Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversioncited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 20; 2026: 7.

27 papers.

Sort:

By area:

  1. 2025CreatiDesign: A Unified Multi-Conditional Diffusion Transformer for Creative Graphic Designcited by 36
  2. 2025SliderSpace: Decomposing the Visual Capabilities of Diffusion Modelscited by 26
  3. 2025RePrompt: Reasoning-Augmented Reprompting for Text-to-Image Generation via Reinforcement Learningcited by 23
  4. 2025GeoDrive: 3D Geometry-Informed Driving World Model with Precise Action Controlcited by 19
  5. 2025WithAnyone: Towards Controllable and ID Consistent Image Generationcited by 17
  6. 2025InstanceAssemble: Layout-Aware Image Generation via Instance Assembling Attentioncited by 10
  7. 2025OverLayBench: A Benchmark for Layout-to-Image Generation with Dense Overlapscited by 10
  8. 2025MultiHuman-Testbench: Benchmarking Image Generation for Multiple Humanscited by 8
  9. 2025RepText: Rendering Visual Text via Replicatingcited by 7
  10. 2026Layer-wise Instance Binding for Regional and Occlusion Control in Text-to-Image Diffusion Transformerscited by 5
  11. 2025PositionIC: Unified Position and Identity Consistency for Image Customizationcited by 4
  12. 2025LORE: Latent Optimization for Precise Semantic Control in Rectified Flow-based Image Editingcited by 3
15 more
  1. 2025RAGSR: Regional Attention Guided Diffusion for Image Super-Resolutioncited by 3
  2. 2025DCText: Scheduled Attention Masking for Visual Text Generation via Divide-and-Conquer Strategycited by 3
  3. 2025CannyEdit: Selective Canny Control and Dual-Prompt Guidance for Training-Free Image Editingcited by 2
  4. 2025DeLeaker: Dynamic Inference-Time Reweighting For Semantic Leakage Mitigation in Text-to-Image Modelscited by 2
  5. 2025DegDiT: Controllable Audio Generation with Dynamic Event Graph Guided Diffusion Transformercited by 1
  6. 2025Taming Identity Consistency and Prompt Diversity in Diffusion Models via Latent Concatenation and Masked Conditional Flow Matchingcited by 1
  7. 2025DreamingComics: A Story Visualization Pipeline via Subject and Layout Customized Generation using Video Modelscited by 1
  8. 2025Dynamic Image Prompt Adapter for Scalable Zero-shot Personalized Text-to-Image Generationcited by 1
  9. 20263D Space as a Scratchpad for Editable Text-to-Image Generationcited by 1
  10. 2026Training-free Motion Factorization for Compositional Video Generationcited by 1
  11. 2026DepthArb: Training-Free Depth-Arbitrated Generation for Occlusion-Robust Image Synthesiscited by 1
  12. 2025SketchAssist: A Practical Assistant for Semantic Edits and Precise Local Redrawingcited by 0
  13. 2026COLLAR: Cascaded Object-Level Latent Refinement for High-Fidelity Conditional Generationcited by 0
  14. 2026Spatially-Grounded Text-to-Video Generation via Inference-Time Gradient-Free Optimizationcited by 0
  15. 2026OverLay++: Dense-Overlap Layout-to-Image Generation Datasetcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models
  2. 2023Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models
  3. 2024Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models
  4. 2025Enhancing Image Generation Fidelity via Progressive Prompts
  5. 2025Exploring Multimodal Diffusion Transformers for Enhanced Prompt-based Image Editing
  6. 2024Seek for Incantations: Towards Accurate Text-to-Image Diffusion Synthesis through Prompt Engineering
  7. 2023Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image Diffusion Models
  8. 2025Decoder-Only LLMs are Better Controllers for Diffusion Models