# LoCo: Locally Constrained Training-Free Layout-to-Image Synthesis

arXiv: https://arxiv.org/abs/2311.12342 · HTML: https://mlsanity.com/paper/2311.12342/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 43 of its 52 references that are in the map.
- First posted on arXiv 21 Nov 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 43)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023)
- [Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2301.13826/) (2023)
- [MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation](https://mlsanity.com/paper/2302.08113/) (2023)
- [Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors](https://mlsanity.com/paper/2203.13131/) (2022)
- [Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis](https://mlsanity.com/paper/2212.05032/) (2022)
- [Training-Free Layout Control with Cross-Attention Guidance](https://mlsanity.com/paper/2304.03373/) (2023)
- [BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion](https://mlsanity.com/paper/2307.10816/) (2023)
- [SpaText: Spatio-Textual Representation for Controllable Image Generation](https://mlsanity.com/paper/2211.14305/) (2022)
- [ReCo: Region-Controlled Text-to-Image Generation](https://mlsanity.com/paper/2211.15518/) (2022)
- [Grounded Text-to-Image Synthesis with Attention Refocusing](https://mlsanity.com/paper/2306.05427/) (2023)
- [Dense Text-to-Image Generation with Attention Modulation](https://mlsanity.com/paper/2308.12964/) (2023)
- [Simple multi-dataset detection](https://mlsanity.com/paper/2102.13086/) (2021)
- [Dataset Diffusion: Diffusion-based Synthetic Dataset Generation for Pixel-Level Semantic Segmentation](https://mlsanity.com/paper/2309.14303/) (2023)
- [Image Generation from Layout](https://mlsanity.com/paper/1811.11389/) (2018)
- [Cones 2: Customizable Image Synthesis with Multiple Subjects](https://mlsanity.com/paper/2305.19327/) (2023)
- [HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Models](https://mlsanity.com/paper/2304.05390/) (2023)
- [Compositional Text-to-Image Synthesis with Attention Map Control of Diffusion Models](https://mlsanity.com/paper/2305.13921/) (2023)

## Built on by (most cited first, 21 of 21)

- [Controllable Generation with Text-to-Image Diffusion Models: A Survey](https://mlsanity.com/paper/2403.04279/) (2024)
- [Be Yourself: Bounded Attention for Multi-Subject Text-to-Image Generation](https://mlsanity.com/paper/2403.16990/) (2024)
- [Rethinking the Spatial Inconsistency in Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2404.05384/) (2024)
- [HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generation](https://mlsanity.com/paper/2410.14324/) (2024)
- [ElasticDiffusion: Training-free Arbitrary Size Image Generation through Global-Local Content Separation](https://mlsanity.com/paper/2311.18822/) (2023)
- [Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesis](https://mlsanity.com/paper/2411.07132/) (2024)
- [Unleashing Unlabeled Data: A Paradigm for Cross-View Geo-Localization](https://mlsanity.com/paper/2403.14198/) (2024)
- [ComfyGPT: A Self-Optimizing Multi-Agent System for Comprehensive ComfyUI Workflow Generation](https://mlsanity.com/paper/2503.17671/) (2025)
- [YOLO-Count: Differentiable Object Counting for Text-to-Image Generation](https://mlsanity.com/paper/2508.00728/) (2025)
- [Multitwine: Multi-Object Compositing with Text and Layout Control](https://mlsanity.com/paper/2502.05165/) (2025)
- [ReGround: Improving Textual and Spatial Grounding at No Cost](https://mlsanity.com/paper/2403.13589/) (2024)
- [Steer Away From Mode Collisions: Improving Composition In Diffusion Models](https://mlsanity.com/paper/2509.25940/) (2025)
- [Training-Free Layout-to-Image Generation with Marginal Attention Constraints](https://mlsanity.com/paper/2411.10495/) (2024)
- [Detail++: Training-Free Detail Enhancer for T2I Diffusion Models](https://mlsanity.com/paper/2507.17853/) (2025)
- [Automatic Scene Generation: State-of-the-Art Techniques, Models, Datasets, Challenges, and Future Prospects](https://mlsanity.com/paper/2410.01816/) (2024)
- [Personalized Image Generation via Human-in-the-loop Bayesian Optimization](https://mlsanity.com/paper/2602.02388/) (2026)
- [HCMA: Hierarchical Cross-model Alignment for Grounded Text-to-Image Generation](https://mlsanity.com/paper/2505.06512/) (2025)
- [MALeR: Improving Compositional Fidelity in Layout-Guided Generation](https://mlsanity.com/paper/2511.06002/) (2025)
- [Appearance Pointers -- Multimodal Region Control of Diffusion Transformers](https://mlsanity.com/paper/2607.19344/) (2026)
- [TILT: Improving Compositional Generation in Diffusion Models with a Model-Intrinsic Reward](https://mlsanity.com/paper/2607.21606/) (2026)
- [OccluRank: Controllable Occlusion-Aware Layout-to-Image Generation by Adding Just an Ordinal Rank](https://mlsanity.com/paper/2608.20932/) (2026)

## Abstract (arXiv)

Recent text-to-image diffusion models have reached an unprecedented level in generating high-quality images. However, their exclusive reliance on textual prompts often falls short in precise control of image compositions. In this paper, we propose LoCo, a training-free approach for layout-to-image Synthesis that excels in producing high-quality images aligned with both textual prompts and layout instructions. Specifically, we introduce a Localized Attention Constraint (LAC), leveraging semantic affinity between pixels in self-attention maps to create precise representations of desired objects and effectively ensure the accurate placement of objects in designated regions. We further propose a Padding Token Constraint (PTC) to leverage the semantic information embedded in previously neglected padding tokens, improving the consistency between object appearance and layout instructions. LoCo seamlessly integrates into existing text-to-image and layout-to-image models, enhancing their performance in spatial control and addressing semantic failures observed in prior methods. Extensive experiments showcase the superiority of our approach, surpassing existing state-of-the-art training-free layout-to-image methods both qualitatively and quantitatively across multiple benchmarks.