# Separate-and-Enhance: Compositional Finetuning for Text2Image Diffusion Models

arXiv: https://arxiv.org/abs/2312.06712 · HTML: https://mlsanity.com/paper/2312.06712/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 42 of its 51 references that are in the map.
- First posted on arXiv 10 Dec 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 42)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022)
- [Vector Quantized Diffusion Model for Text-to-Image Synthesis](https://mlsanity.com/paper/2111.14822/) (2021)
- [Generative Adversarial Text to Image Synthesis](https://mlsanity.com/paper/1605.05396/) (2016)
- [Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2301.13826/) (2023)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)
- [Compositional Visual Generation with Composable Diffusion Models](https://mlsanity.com/paper/2206.01714/) (2022)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1612.03242/) (2016)
- [Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis](https://mlsanity.com/paper/2212.05032/) (2022)
- [DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesis](https://mlsanity.com/paper/1904.01310/) (2019)
- [Diffusion Self-Guidance for Controllable Image Generation](https://mlsanity.com/paper/2306.00986/) (2023)

## Built on by (most cited first, 19 of 19)

- [FlowMo: Variance-Based Flow Guidance for Coherent Motion in Video Generation](https://mlsanity.com/paper/2506.01144/) (2025)
- [Self-Cross Diffusion Guidance for Text-to-Image Synthesis of Similar Subjects](https://mlsanity.com/paper/2411.18936/) (2024)
- [Object-level Visual Prompts for Compositional Image Generation](https://mlsanity.com/paper/2501.01424/) (2025)
- [Towards Transformer-Based Aligned Generation with Self-Coherence Guidance](https://mlsanity.com/paper/2503.17675/) (2025)
- [Enhancing MMDiT-Based Text-to-Image Models for Similar Subject Generation](https://mlsanity.com/paper/2411.18301/) (2024)
- [Be Decisive: Noise-Induced Layouts for Multi-Subject Generation](https://mlsanity.com/paper/2505.21488/) (2025)
- [YOLO-Count: Differentiable Object Counting for Text-to-Image Generation](https://mlsanity.com/paper/2508.00728/) (2025)
- [Stepwise Credit Assignment for GRPO on Flow-Matching Models](https://mlsanity.com/paper/2603.28718/) (2026)
- [Diffusion-based Visual Anagram as Multi-task Learning](https://mlsanity.com/paper/2412.02693/) (2024)
- [Steer Away From Mode Collisions: Improving Composition In Diffusion Models](https://mlsanity.com/paper/2509.25940/) (2025)
- [Understanding Multi-Granularity for Open-Vocabulary Part Segmentation](https://mlsanity.com/paper/2406.11384/) (2024)
- [Generative Photomontage](https://mlsanity.com/paper/2408.07116/) (2024)
- [Cert-LAS: Toward Certified Model Ownership Verification for Text-to-Image Diffusion Models via Layer-Adaptive Smoothing](https://mlsanity.com/paper/2605.29809/) (2026)
- [Scribble-Guided Diffusion for Training-free Text-to-Image Generation](https://mlsanity.com/paper/2409.08026/) (2024)
- [AGSwap: Overcoming Category Boundaries in Object Fusion via Adaptive Group Swapping](https://mlsanity.com/paper/2509.18699/) (2025)
- [Improving Compositional Generation with Diffusion Models Using Lift Scores](https://mlsanity.com/paper/2505.13740/) (2025)
- [PICS: Pairwise Image Compositing with Spatial Interactions](https://mlsanity.com/paper/2603.06873/) (2026)
- [Training-Free Object-Background Compositional T2I via Dynamic Spatial Guidance and Multi-Path Pruning](https://mlsanity.com/paper/2604.09850/) (2026)
- [TeDiO: Temporal Diagonal Optimization for Training-Free Coherent Video Diffusion](https://mlsanity.com/paper/2605.14136/) (2026)

## Abstract (arXiv)

Despite recent significant strides achieved by diffusion-based Text-to-Image (T2I) models, current systems are still less capable of ensuring decent compositional generation aligned with text prompts, particularly for the multi-object generation. This work illuminates the fundamental reasons for such misalignment, pinpointing issues related to low attention activation scores and mask overlaps. While previous research efforts have individually tackled these issues, we assert that a holistic approach is paramount. Thus, we propose two novel objectives, the Separate loss and the Enhance loss, that reduce object mask overlaps and maximize attention scores, respectively. Our method diverges from conventional test-time-adaptation techniques, focusing on finetuning critical parameters, which enhances scalability and generalizability. Comprehensive evaluations demonstrate the superior performance of our model in terms of image realism, text-image alignment, and adaptability, notably outperforming prominent baselines. Ultimately, this research paves the way for T2I diffusion models with enhanced compositional capacities and broader applicability.