# SpotActor: Training-Free Layout-Controlled Consistent Image Generation arXiv: https://arxiv.org/abs/2409.04801 ยท HTML: https://mlsanity.com/paper/2409.04801/ ## Key facts - Cited by 12 papers in the map (data through 14 Oct 2026). - Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. - Builds on 23 of its 30 references that are in the map. - First posted on arXiv 7 Sep 2024. ## Placed in - Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/) ## Builds on (most cited first, 23 of 23) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020) - [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018) - [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020) - [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015) - [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019) - [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022) - [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022) - [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021) - [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022) - [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022) - [PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding](https://mlsanity.com/paper/2312.04461/) (2023) - [Training-Free Layout Control with Cross-Attention Guidance](https://mlsanity.com/paper/2304.03373/) (2023) - [Diffusion Self-Guidance for Controllable Image Generation](https://mlsanity.com/paper/2306.00986/) (2023) - [BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion](https://mlsanity.com/paper/2307.10816/) (2023) - [StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation](https://mlsanity.com/paper/2405.01434/) (2024) - [Dense Text-to-Image Generation with Attention Modulation](https://mlsanity.com/paper/2308.12964/) (2023) - [FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Model with Any Condition](https://mlsanity.com/paper/2312.07536/) (2023) - [Make-A-Story: Visual Memory Conditioned Consistent Story Generation](https://mlsanity.com/paper/2211.13319/) (2022) - [OneActor: Consistent Character Generation via Cluster-Conditioned Guidance](https://mlsanity.com/paper/2404.10267/) (2024) ## Built on by (most cited first, 12 of 12) - [EchoShot: Multi-Shot Portrait Video Generation](https://mlsanity.com/paper/2506.15838/) (2025) - [LaRender: Training-Free Occlusion Control in Image Generation via Latent Rendering](https://mlsanity.com/paper/2508.07647/) (2025) - [DynamicID: Zero-Shot Multi-ID Image Personalization with Flexible Facial Editability](https://mlsanity.com/paper/2503.06505/) (2025) - [CC-Diff: Enhancing Contextual Coherence in Remote Sensing Image Synthesis](https://mlsanity.com/paper/2412.08464/) (2024) - [PlanViz: Evaluating Planning-Oriented Image Generation and Editing for Computer-Use Tasks](https://mlsanity.com/paper/2602.06663/) (2026) - [MotionWeaver: Holistic 4D-Anchored Framework for Multi-Humanoid Image Animation](https://mlsanity.com/paper/2602.13326/) (2026) - [StorySync: Training-Free Subject Consistency in Text-to-Image Generation via Region Harmonization](https://mlsanity.com/paper/2508.03735/) (2025) - [FontCrafter: High-Fidelity Element-Driven Artistic Font Creation with Visual In-Context Generation](https://mlsanity.com/paper/2603.22054/) (2026) - [DepthArb: Training-Free Depth-Arbitrated Generation for Occlusion-Robust Image Synthesis](https://mlsanity.com/paper/2603.23924/) (2026) - [AnyID: Ultra-Fidelity Universal Identity-Preserving Video Generation from Any Visual References](https://mlsanity.com/paper/2603.25188/) (2026) - [A Training-Free Approach for Multi-ID Customization via Attention Adjustment and Spatial Control](https://mlsanity.com/paper/2511.20401/) (2025) - [DEIG: Detail-Enhanced Instance Generation with Fine-Grained Semantic Control](https://mlsanity.com/paper/2602.18282/) (2026) ## Abstract (arXiv) Text-to-image diffusion models significantly enhance the efficiency of artistic creation with high-fidelity image generation. However, in typical application scenarios like comic book production, they can neither place each subject into its expected spot nor maintain the consistent appearance of each subject across images. For these issues, we pioneer a novel task, Layout-to-Consistent-Image (L2CI) generation, which produces consistent and compositional images in accordance with the given layout conditions and text prompts. To accomplish this challenging task, we present a new formalization of dual energy guidance with optimization in a dual semantic-latent space and thus propose a training-free pipeline, SpotActor, which features a layout-conditioned backward update stage and a consistent forward sampling stage. In the backward stage, we innovate a nuanced layout energy function to mimic the attention activations with a sigmoid-like objective. While in the forward stage, we design Regional Interconnection Self-Attention (RISA) and Semantic Fusion Cross-Attention (SFCA) mechanisms that allow mutual interactions across images. To evaluate the performance, we present ActorBench, a specified benchmark with hundreds of reasonable prompt-box pairs stemming from object detection datasets. Comprehensive experiments are conducted to demonstrate the effectiveness of our method. The results prove that SpotActor fulfills the expectations of this task and showcases the potential for practical applications with superior layout alignment, subject consistency, prompt conformity and background diversity.