# Layout Control and Semantic Guidance with Attention Loss Backward for T2I Diffusion Model

arXiv: https://arxiv.org/abs/2411.06692 · HTML: https://mlsanity.com/paper/2411.06692/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 12 of its 12 references that are in the map.
- First posted on arXiv 11 Nov 2024.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 12 of 12)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023)
- [Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2301.13826/) (2023)
- [Training-Free Layout Control with Cross-Attention Guidance](https://mlsanity.com/paper/2304.03373/) (2023)
- [ReCo: Region-Controlled Text-to-Image Generation](https://mlsanity.com/paper/2211.15518/) (2022)
- [Smartbanner: Intelligent banner design framework that strikes a balance between creative freedom and design rules](https://mlsanity.com/paper/2303.10325/) (2023)
- [Two-stage dynamic creative optimization under sparse ambiguous samples for e-commerce advertising](https://mlsanity.com/paper/2312.01295/) (2023)
- [E-Commerce Inpainting with Mask Guidance in Controlnet for Reducing Overcompletion](https://mlsanity.com/paper/2409.09681/) (2024)
- [Training-Free Style Consistent Image Synthesis with Condition and Mask Guidance in E-Commerce](https://mlsanity.com/paper/2409.04750/) (2024)
- [Intelligent Parsing: An Automated Parsing Framework for Extracting Design Semantics from E-commerce Creatives](https://mlsanity.com/paper/2312.17283/) (2023)

## Built on by (most cited first, 5 of 5)

- [Dual-Channel Attention Guidance for Training-Free Image Editing Control in Diffusion Transformers](https://mlsanity.com/paper/2602.18022/) (2026)
- [DVI: Disentangling Semantic and Visual Identity for Training-Free Personalized Generation](https://mlsanity.com/paper/2512.18964/) (2025)
- [FlexID: Training-Free Flexible Identity Injection via Intent-Aware Modulation for Text-to-Image Generation](https://mlsanity.com/paper/2602.07554/) (2026)
- [Inject Where It Matters: Training-Free Spatially-Adaptive Identity Preservation for Text-to-Image Personalization](https://mlsanity.com/paper/2602.13994/) (2026)
- [AdaEdit: Adaptive Temporal and Channel Modulation for Flow-Based Image Editing](https://mlsanity.com/paper/2603.21615/) (2026)

## Abstract (arXiv)

Controllable image generation has always been one of the core demands in image generation, aiming to create images that are both creative and logical while satisfying additional specified conditions. In the post-AIGC era, controllable generation relies on diffusion models and is accomplished by maintaining certain components or introducing inference interferences. This paper addresses key challenges in controllable generation: 1. mismatched object attributes during generation and poor prompt-following effects; 2. inadequate completion of controllable layouts. We propose a train-free method based on attention loss backward, cleverly controlling the cross attention map. By utilizing external conditions such as prompts that can reasonably map onto the attention map, we can control image generation without any training or fine-tuning. This method addresses issues like attribute mismatch and poor prompt-following while introducing explicit layout constraints for controllable image generation. Our approach has achieved excellent practical applications in production, and we hope it can serve as an inspiring technical report in this field.