mlsanityData to 7 Oct 2026

BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion

arXiv 2307.10816 ↗First posted 20 Jul 2023cs.CVcited by 280

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Recent text-to-image diffusion models have demonstrated an astonishing capacity to generate high-quality images. However, researchers mainly studied the way of synthesizing images with only text prompts. While some works have explored using other modalities as conditions, considerable paired data, e.g., box/mask-image pairs, and fine-tuning time are required for nurturing models. As such paired data is time-consuming and labor-intensive to acquire and restricted to a closed set, this potentially becomes the bottleneck for applications in an open world. This paper focuses on the simplest form of user-provided conditions, e.g., box or scribble. To mitigate the aforementioned problem, we propose a training-free method to control objects and contexts in the synthesized images adhering to the given spatial conditions. Specifically, three spatial constraints, i.e., Inner-Box, Outer-Box, and Corner Constraints, are designed and seamlessly integrated into the denoising step of diffusion models, requiring no additional training and massive annotated layout data. Extensive experimental results demonstrate that the proposed constraints can control what and where to present in the images while retaining the ability of Diffusion models to synthesize with high fidelity and diverse concept coverage. The code is publicly available at https://github.com/showlab/BoxDiff.

Key facts. Cited by 280 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 38 of its 42 references that are in the map. First posted on arXiv 20 Jul 2023.

Builds on

38 of 42 references.

Closest ancestors · 14

Same direction, newest first

  1. 2023Grounded Text-to-Image Synthesis with Attention Refocusing8 Jun
  2. 2023Controllable Text-to-Image Generation with GPT-429 May
  3. 2023Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models29 May
  4. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models23 May
  5. 2023Directed Diffusion: Direct Control of Object Placement through Attention Guidance25 Feb
  6. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  7. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models31 Jan
  8. 2023GLIGEN: Open-Set Grounded Text-to-Image Generation17 Jan
  9. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis9 Dec
  10. 2022SpaText: Spatio-Textual Representation for Controllable Image Generation25 Nov
  11. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
  12. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
2 more
  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr

Other references · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  10. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  11. 2020YOLOv4: Optimal Speed and Accuracy of Object Detectioncited by 1,624
  12. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
12 more
  1. 2021ILVR: Conditioning Method for Denoising Diffusion Probabilistic Modelscited by 634
  2. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  3. 2023Universal Guidance for Diffusion Modelscited by 426
  4. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  5. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 312
  6. 2023Key-Locked Rank One Editing for Text-to-Image Personalizationcited by 194
  7. 2022ReCo: Region-Controlled Text-to-Image Generationcited by 177
  8. 2018Image Generation from Layoutcited by 105
  9. 2020Object-Centric Image Generation from Layoutscited by 57
  10. 2020Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesiscited by 49
  11. 2022Modeling Image Composition for Complex Scene Generationcited by 43
  12. 2023VisorGPT: Learning Visual Prior via Generative Pre-Trainingcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 22; 2024: 116; 2025: 87; 2026: 55.

280 papers.

Sort:

By area:

  1. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  2. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 455
  3. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
  4. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  5. 2024InstanceDiffusion: Instance-level Control for Image Generationcited by 175
  6. 2024ControlNet++: Improving Conditional Controls with Efficient Consistency Feedbackcited by 153
  7. 2024GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editingcited by 132
  8. 2024MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesiscited by 123
  9. 2024InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimizationcited by 123
  10. 2023Self-correcting LLM-controlled Diffusion Modelscited by 112
  11. 2024EraseAnything: Enabling Concept Erasure in Rectified Flow Transformerscited by 93
  12. 2024MotionBooth: Motion-Aware Customized Text-to-Video Generationcited by 92
38 more
  1. 2023TrailBlazer: Trajectory Control for Diffusion-Based Video Generationcited by 90
  2. 2023FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Model with Any Conditioncited by 89
  3. 2023LLM-grounded Video Diffusion Modelscited by 83
  4. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 82
  5. 2024Be Yourself: Bounded Attention for Multi-Subject Text-to-Image Generationcited by 68
  6. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?cited by 64
  7. 2023Focus on Your Instruction: Fine-grained and Multi-instruction Image Editing by Attention Modulationcited by 61
  8. 2024CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matchingcited by 60
  9. 2023Ground-A-Video: Zero-shot Grounded Video Editing using Text-to-image Diffusion Modelscited by 58
  10. 2024Rethinking the Spatial Inconsistency in Classifier-Free Diffusion Guidancecited by 58
  11. 2024LoRA-Composer: Leveraging Low-Rank Adaptation for Multi-Concept Customization in Training-Free Diffusion Modelscited by 55
  12. 2023ControlRoom3D: Room Generation using Semantic Proxy Roomscited by 54
  13. 2024Faster Diffusion via Temporal Attention Decompositioncited by 54
  14. 2024T2V-Turbo-v2: Enhancing Video Generation Model Post-Training through Data, Reward, and Conditional Guidance Designcited by 54
  15. 2024VideoTetris: Towards Compositional Text-to-Video Generationcited by 51
  16. 2024Diff-Plugin: Revitalizing Details for Diffusion-based Low-level Taskscited by 47
  17. 2024StyleMaster: Stylize Your Video with Artistic Generation and Translationcited by 47
  18. 2024MultiBooth: Towards Generating All Your Concepts in an Image from Textcited by 45
  19. 2025Reflect-DiT: Inference-Time Scaling for Text-to-Image Diffusion Transformers via In-Context Reflectioncited by 42
  20. 2024HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generationcited by 39
  21. 2023AutoStory: Generating Diverse Storytelling Images with Minimal Human Effortcited by 38
  22. 2024Region-Aware Text-to-Image Generation via Hard Binding and Soft Refinementcited by 37
  23. 2023ArtBank: Artistic Style Transfer with Pre-trained Diffusion Model and Implicit Style Prompt Bankcited by 36
  24. 20243DIS: Depth-Driven Decoupled Instance Synthesis for Text-to-Image Generationcited by 36
  25. 2023MosaicFusion: Diffusion Models as Data Augmenters for Large Vocabulary Instance Segmentationcited by 35
  26. 2024MIGC++: Advanced Multi-Instance Generation Controller for Image Synthesiscited by 35
  27. 2024Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesiscited by 35
  28. 2024GeoSynth: Contextually-Aware High-Resolution Satellite Image Synthesiscited by 32
  29. 2024IterComp: Iterative Composition-Aware Feedback Learning from Model Gallery for Text-to-Image Generationcited by 32
  30. 2024Neural Assets: 3D-Aware Multi-Object Scene Synthesis with Image Diffusion Modelscited by 30
  31. 2023R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generationcited by 29
  32. 2024Learning Flow Fields in Attention for Controllable Person Image Generationcited by 27
  33. 2024SeaS: Few-shot Industrial Anomaly Image Generation with Separation and Sharing Fine-tuningcited by 26
  34. 2024Motion-Zero: Zero-Shot Moving Object Control Framework for Diffusion-Based Video Generationcited by 25
  35. 2024NoiseCollage: A Layout-Aware Text-to-Image Diffusion Model Based on Noise Cropping and Mergingcited by 25
  36. 2024FreeTuner: Any Subject in Any Style with Training-free Diffusioncited by 24
  37. 2024GrounDiT: Grounding Diffusion Transformers via Noisy Patch Transplantationcited by 24
  38. 2025MCCD: Multi-Agent Collaboration-based Compositional Diffusion for Complex Text-to-Image Generationcited by 23

50 most cited of 280 All 280 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Conditional Image Synthesis with Diffusion Models: A Survey
  2. 2021More Control for Free! Image Synthesis with Semantic Diffusion Guidance
  3. 2024Fast constrained sampling in pre-trained diffusion models
  4. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation
  5. 2024Test-time Conditional Text-to-Image Synthesis Using Diffusion Models
  6. 2022Unite and Conquer: Plug & Play Multi-Modal Synthesis using Diffusion Models
  7. 2023Training-free Diffusion Model Adaptation for Variable-Sized Text-to-Image Synthesis
  8. 2023Modulating Pretrained Diffusion Models for Multimodal Image Synthesis