# Localized Text-to-Image Generation for Free via Cross Attention Control

arXiv: https://arxiv.org/abs/2306.14636 · HTML: https://mlsanity.com/paper/2306.14636/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 39 references that are in the map.
- First posted on arXiv 26 Jun 2023.

## Builds on (most cited first, 30 of 31)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023)
- [Compositional Visual Generation with Composable Diffusion Models](https://mlsanity.com/paper/2206.01714/) (2022)
- [Designing an Encoder for StyleGAN Image Manipulation](https://mlsanity.com/paper/2102.02766/) (2021)
- [MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation](https://mlsanity.com/paper/2302.08113/) (2023)
- [Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality](https://mlsanity.com/paper/2204.03162/) (2022)
- [Scaling up GANs for Text-to-Image Synthesis](https://mlsanity.com/paper/2303.05511/) (2023)
- [Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis](https://mlsanity.com/paper/2212.05032/) (2022)
- [Image Generation from Layout](https://mlsanity.com/paper/1811.11389/) (2018)

## Built on by (most cited first, 27 of 27)

- [Dense Text-to-Image Generation with Attention Modulation](https://mlsanity.com/paper/2308.12964/) (2023)
- [Controllable Generation with Text-to-Image Diffusion Models: A Survey](https://mlsanity.com/paper/2403.04279/) (2024)
- [Be Yourself: Bounded Attention for Multi-Subject Text-to-Image Generation](https://mlsanity.com/paper/2403.16990/) (2024)
- [Fair Text-to-Image Diffusion via Fair Mapping](https://mlsanity.com/paper/2311.17695/) (2023)
- [ControlNet-XS: Rethinking the Control of Text-to-Image Diffusion Models as Feedback-Control Systems](https://mlsanity.com/paper/2312.06573/) (2023)
- [LoCo: Locally Constrained Training-Free Layout-to-Image Synthesis](https://mlsanity.com/paper/2311.12342/) (2023)
- [Object-level Visual Prompts for Compositional Image Generation](https://mlsanity.com/paper/2501.01424/) (2025)
- [Automated Black-box Prompt Engineering for Personalized Text-to-Image Generation](https://mlsanity.com/paper/2403.19103/) (2024)
- [Scaling Group Inference for Diverse and High-Quality Generation](https://mlsanity.com/paper/2508.15773/) (2025)
- [LoMOE: Localized Multi-Object Editing via Multi-Diffusion](https://mlsanity.com/paper/2403.00437/) (2024)
- [InstanceAssemble: Layout-Aware Image Generation via Instance Assembling Attention](https://mlsanity.com/paper/2509.16691/) (2025)
- [Layered Rendering Diffusion Model for Controllable Zero-Shot Image Synthesis](https://mlsanity.com/paper/2311.18435/) (2023)
- [Towards Better Text-to-Image Generation Alignment via Attention Modulation](https://mlsanity.com/paper/2404.13899/) (2024)
- [Personalized Residuals for Concept-Driven Text-to-Image Generation](https://mlsanity.com/paper/2405.12978/) (2024)
- [ROICtrl: Boosting Instance Control for Visual Generation](https://mlsanity.com/paper/2411.17949/) (2024)
- [Multi-modal Generative AI: Multi-modal LLMs, Diffusions, and the Unification](https://mlsanity.com/paper/2409.14993/) (2024)
- [Enhancing Object Coherence in Layout-to-Image Synthesis](https://mlsanity.com/paper/2311.10522/) (2023)
- [AttenCraft: Attention-guided Disentanglement of Multiple Concepts for Text-to-Image Customization](https://mlsanity.com/paper/2405.17965/) (2024)
- [Layout-to-Image Generation with Localized Descriptions using ControlNet with Cross-Attention Control](https://mlsanity.com/paper/2402.13404/) (2024)
- [DreamBeast: Distilling 3D Fantastical Animals with Part-Aware Knowledge Transfer](https://mlsanity.com/paper/2409.08271/) (2024)
- [Concept Conductor: Orchestrating Multiple Personalized Concepts in Text-to-Image Synthesis](https://mlsanity.com/paper/2408.03632/) (2024)
- [TTOM: Test-Time Optimization and Memorization for Compositional Video Generation](https://mlsanity.com/paper/2510.07940/) (2025)
- [RealCraft: Attention Control as A Tool for Zero-Shot Consistent Video Editing](https://mlsanity.com/paper/2312.12635/) (2023)
- [Textual Localization: Decomposing Multi-concept Images for Subject-Driven Text-to-Image Generation](https://mlsanity.com/paper/2402.09966/) (2024)
- [LTOS: Layout-controllable Text-Object Synthesis via Adaptive Cross-attention Fusions](https://mlsanity.com/paper/2404.13579/) (2024)
- [Compositional Video Generation via Inference-Time Guidance](https://mlsanity.com/paper/2605.14988/) (2026)
- [Appearance Pointers -- Multimodal Region Control of Diffusion Transformers](https://mlsanity.com/paper/2607.19344/) (2026)

## Abstract (arXiv)

Despite the tremendous success in text-to-image generative models, localized text-to-image generation (that is, generating objects or features at specific locations in an image while maintaining a consistent overall generation) still requires either explicit training or substantial additional inference time. In this work, we show that localized generation can be achieved by simply controlling cross attention maps during inference. With no additional training, model architecture modification or inference time, our proposed cross attention control (CAC) provides new open-vocabulary localization abilities to standard text-to-image models. CAC also enhances models that are already trained for localized generation when deployed at inference time. Furthermore, to assess localized text-to-image generation performance automatically, we develop a standardized suite of evaluations using large pretrained recognition models. Our experiments show that CAC improves localized generation performance with various types of location information ranging from bounding boxes to semantic segmentation maps, and enhances the compositional capability of state-of-the-art text-to-image generative models.