# R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation

arXiv: https://arxiv.org/abs/2310.08872 · HTML: https://mlsanity.com/paper/2310.08872/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 39 of its 45 references that are in the map.
- First posted on arXiv 13 Oct 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 39)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023)
- [Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2301.13826/) (2023)
- [Emergent Correspondence from Image Diffusion](https://mlsanity.com/paper/2306.03881/) (2023)
- [Universal Guidance for Diffusion Models](https://mlsanity.com/paper/2302.07121/) (2023)
- [Training-Free Layout Control with Cross-Attention Guidance](https://mlsanity.com/paper/2304.03373/) (2023)
- [Diffusion Self-Guidance for Controllable Image Generation](https://mlsanity.com/paper/2306.00986/) (2023)
- [Perception Prioritized Training of Diffusion Models](https://mlsanity.com/paper/2204.00227/) (2022)
- [BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion](https://mlsanity.com/paper/2307.10816/) (2023)
- [LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generation](https://mlsanity.com/paper/2303.17189/) (2023)
- [LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models](https://mlsanity.com/paper/2305.13655/) (2023)
- [SpaText: Spatio-Textual Representation for Controllable Image Generation](https://mlsanity.com/paper/2211.14305/) (2022)
- [DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Models](https://mlsanity.com/paper/2307.02421/) (2023)
- [Sketch-Guided Text-to-Image Diffusion Models](https://mlsanity.com/paper/2211.13752/) (2022)
- [Grounded Text-to-Image Synthesis with Attention Refocusing](https://mlsanity.com/paper/2306.05427/) (2023)

## Built on by (most cited first, 29 of 29)

- [Controllable Generation with Text-to-Image Diffusion Models: A Survey](https://mlsanity.com/paper/2403.04279/) (2024)
- [3DIS: Depth-Driven Decoupled Instance Synthesis for Text-to-Image Generation](https://mlsanity.com/paper/2410.12669/) (2024)
- [Ctrl-X: Controlling Structure and Appearance for Text-To-Image Generation Without Guidance](https://mlsanity.com/paper/2406.07540/) (2024)
- [DreamRenderer: Taming Multi-Instance Attribute Control in Large-Scale Text-to-Image Models](https://mlsanity.com/paper/2503.12885/) (2025)
- [NoiseCollage: A Layout-Aware Text-to-Image Diffusion Model Based on Noise Cropping and Merging](https://mlsanity.com/paper/2403.03485/) (2024)
- [GrounDiT: Grounding Diffusion Transformers via Noisy Patch Transplantation](https://mlsanity.com/paper/2410.20474/) (2024)
- [LoCo: Locally Constrained Training-Free Layout-to-Image Synthesis](https://mlsanity.com/paper/2311.12342/) (2023)
- [PLACE: Adaptive Layout-Semantic Fusion for Semantic Image Synthesis](https://mlsanity.com/paper/2403.01852/) (2024)
- [IFAdapter: Instance Feature Control for Grounded Text-to-Image Generation](https://mlsanity.com/paper/2409.08240/) (2024)
- [Psi-Sampler: Initial Particle Sampling for SMC-Based Inference-Time Reward Alignment in Score Models](https://mlsanity.com/paper/2506.01320/) (2025)
- [GRA: Detecting Oriented Objects through Group-wise Rotating and Attention](https://mlsanity.com/paper/2403.11127/) (2024)
- [Pick-and-Draw: Training-free Semantic Guidance for Text-to-Image Personalization](https://mlsanity.com/paper/2401.16762/) (2024)
- [Rethinking The Training And Evaluation of Rich-Context Layout-to-Image Generation](https://mlsanity.com/paper/2409.04847/) (2024)
- [CoMPaSS: Enhancing Spatial Understanding in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2412.13195/) (2024)
- [ReGround: Improving Textual and Spatial Grounding at No Cost](https://mlsanity.com/paper/2403.13589/) (2024)
- [Distractors-Immune Representation Learning with Cross-modal Contrastive Regularization for Change Captioning](https://mlsanity.com/paper/2407.11683/) (2024)
- [Training-free Dense-Aligned Diffusion Guidance for Modular Conditional Image Synthesis](https://mlsanity.com/paper/2504.01515/) (2025)
- [Layout-to-Image Generation with Localized Descriptions using ControlNet with Cross-Attention Control](https://mlsanity.com/paper/2402.13404/) (2024)
- [Faster Image2Video Generation: A Closer Look at CLIP Image Embedding's Impact on Spatio-Temporal Cross-Attentions](https://mlsanity.com/paper/2407.19205/) (2024)
- [Training-Free Layout-to-Image Generation with Marginal Attention Constraints](https://mlsanity.com/paper/2411.10495/) (2024)
- [Control and Realism: Best of Both Worlds in Layout-to-Image without Training](https://mlsanity.com/paper/2506.15563/) (2025)
- [Making Training-Free Diffusion Segmentors Scale with the Generative Power](https://mlsanity.com/paper/2603.06178/) (2026)
- [Transferability Between Understanding and Generation in Unified Multimodal Models](https://mlsanity.com/paper/2607.04423/) (2026)
- [Training-free Motion Factorization for Compositional Video Generation](https://mlsanity.com/paper/2603.09104/) (2026)
- [DepthArb: Training-Free Depth-Arbitrated Generation for Occlusion-Robust Image Synthesis](https://mlsanity.com/paper/2603.23924/) (2026)
- [ComposeAnyone: Controllable Layout-to-Human Generation with Decoupled Multimodal Conditions](https://mlsanity.com/paper/2501.12173/) (2025)
- [ISAC: Training-Free Instance-to-Semantic Attention Control for Multi-Instance Generation](https://mlsanity.com/paper/2505.20935/) (2025)
- [RichControl: Structure- and Appearance-Rich Training-Free Spatial Control for Text-to-Image Generation](https://mlsanity.com/paper/2507.02792/) (2025)
- [Appearance Pointers -- Multimodal Region Control of Diffusion Transformers](https://mlsanity.com/paper/2607.19344/) (2026)

## Abstract (arXiv)

Recent text-to-image (T2I) diffusion models have achieved remarkable progress in generating high-quality images given text-prompts as input. However, these models fail to convey appropriate spatial composition specified by a layout instruction. In this work, we probe into zero-shot grounded T2I generation with diffusion models, that is, generating images corresponding to the input layout information without training auxiliary modules or finetuning diffusion models. We propose a Region and Boundary (R&B) aware cross-attention guidance approach that gradually modulates the attention maps of diffusion model during generative process, and assists the model to synthesize images (1) with high fidelity, (2) highly compatible with textual input, and (3) interpreting layout instructions accurately. Specifically, we leverage the discrete sampling to bridge the gap between consecutive attention maps and discrete layout constraints, and design a region-aware loss to refine the generative layout during diffusion process. We further propose a boundary-aware loss to strengthen object discriminability within the corresponding regions. Experimental results show that our method outperforms existing state-of-the-art zero-shot grounded T2I generation methods by a large margin both qualitatively and quantitatively on several benchmarks.