# Progressive Text-to-Image Diffusion with Soft Latent Direction arXiv: https://arxiv.org/abs/2309.09466 ยท HTML: https://mlsanity.com/paper/2309.09466/ ## Key facts - Cited by 6 papers in the map (data through 14 Oct 2026). - Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. - Builds on 35 of its 40 references that are in the map. - First posted on arXiv 18 Sep 2023. ## Placed in - Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/) ## Builds on (most cited first, 30 of 35) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022) - [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022) - [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016) - [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021) - [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022) - [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021) - [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022) - [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022) - [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019) - [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017) - [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022) - [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022) - [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017) - [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021) - [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018) - [Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation](https://mlsanity.com/paper/2008.00951/) (2020) - [Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2301.13826/) (2023) - [Unsupervised Image-to-Image Translation Networks](https://mlsanity.com/paper/1703.00848/) (2017) - [Paint by Example: Exemplar-based Image Editing with Diffusion Models](https://mlsanity.com/paper/2211.13227/) (2022) - [Blended Latent Diffusion](https://mlsanity.com/paper/2206.02779/) (2022) - [In-Domain GAN Inversion for Real Image Editing](https://mlsanity.com/paper/2004.00049/) (2020) - [Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis](https://mlsanity.com/paper/2212.05032/) (2022) - [Generative Visual Manipulation on the Natural Image Manifold](https://mlsanity.com/paper/1609.03552/) (2016) - [SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Model](https://mlsanity.com/paper/2212.05034/) (2022) - [Paint by Word](https://mlsanity.com/paper/2103.10951/) (2021) - [Scribbler: Controlling Deep Image Synthesis with Sketch and Color](https://mlsanity.com/paper/1612.00835/) (2016) - [Transforming and Projecting Images into Class-conditional Generative Networks](https://mlsanity.com/paper/2005.01703/) (2020) ## Built on by (most cited first, 6 of 6) - [RealCompo: Balancing Realism and Compositionality Improves Text-to-Image Diffusion Models](https://mlsanity.com/paper/2402.12908/) (2024) - [Video Anomaly Detection with Motion and Appearance Guided Patch Diffusion Model](https://mlsanity.com/paper/2412.09026/) (2024) - [When Priors Backfire: On the Vulnerability of Unlearnable Examples to Pretraining](https://mlsanity.com/paper/2603.04731/) (2026) - [Fine-Grained Alignment and Noise Refinement for Compositional Text-to-Image Generation](https://mlsanity.com/paper/2503.06506/) (2025) - [Full spectrum Unlearnable Examples via Spectral Equalization](https://mlsanity.com/paper/2606.26719/) (2026) - [RankT2I: A Submodular Framework for Discovering Interpretable and Diverse Semantics in Text-to-Image Models](https://mlsanity.com/paper/2608.14226/) (2026) ## Abstract (arXiv) In spite of the rapidly evolving landscape of text-to-image generation, the synthesis and manipulation of multiple entities while adhering to specific relational constraints pose enduring challenges. This paper introduces an innovative progressive synthesis and editing operation that systematically incorporates entities into the target image, ensuring their adherence to spatial and relational constraints at each sequential step. Our key insight stems from the observation that while a pre-trained text-to-image diffusion model adeptly handles one or two entities, it often falters when dealing with a greater number. To address this limitation, we propose harnessing the capabilities of a Large Language Model (LLM) to decompose intricate and protracted text descriptions into coherent directives adhering to stringent formats. To facilitate the execution of directives involving distinct semantic operations-namely insertion, editing, and erasing-we formulate the Stimulus, Response, and Fusion (SRF) framework. Within this framework, latent regions are gently stimulated in alignment with each operation, followed by the fusion of the responsive latent components to achieve cohesive entity manipulation. Our proposed framework yields notable advancements in object synthesis, particularly when confronted with intricate and lengthy textual inputs. Consequently, it establishes a new benchmark for text-to-image generation tasks, further elevating the field's performance standards.