# Collage Diffusion

arXiv: https://arxiv.org/abs/2303.00262 · HTML: https://mlsanity.com/paper/2303.00262/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 30 of its 44 references that are in the map.
- First posted on arXiv 1 Mar 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation](https://mlsanity.com/paper/2211.12572/) (2022)
- [Paint by Example: Exemplar-based Image Editing with Diffusion Models](https://mlsanity.com/paper/2211.13227/) (2022)
- [Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery](https://mlsanity.com/paper/2302.03668/) (2023)
- [Text2LIVE: Text-Driven Layered Image and Video Editing](https://mlsanity.com/paper/2204.02491/) (2022)
- [SpaText: Spatio-Textual Representation for Controllable Image Generation](https://mlsanity.com/paper/2211.14305/) (2022)
- [Layered Neural Atlases for Consistent Video Editing](https://mlsanity.com/paper/2109.11418/) (2021)
- [DoveNet: Deep Image Harmonization via Domain Verification](https://mlsanity.com/paper/1911.13239/) (2019)
- [BargainNet: Background-Guided Domain Translation for Image Harmonization](https://mlsanity.com/paper/2009.09169/) (2020)
- [Layered Neural Rendering for Retiming People in Video](https://mlsanity.com/paper/2009.07833/) (2020)
- [DCCF: Deep Comprehensible Color Filter Learning Framework for High-Resolution Image Harmonization](https://mlsanity.com/paper/2207.04788/) (2022)
- [ObjectStitch: Generative Object Compositing](https://mlsanity.com/paper/2212.00932/) (2022)
- [Shadow Generation for Composite Image in Real-world Scenes](https://mlsanity.com/paper/2104.10338/) (2021)
- [Text-driven Visual Synthesis with Latent Diffusion Prior](https://mlsanity.com/paper/2302.08510/) (2023)

## Built on by (most cited first, 30 of 33)

- [AnyDoor: Zero-shot Object-level Image Customization](https://mlsanity.com/paper/2307.09481/) (2023)
- [FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attention](https://mlsanity.com/paper/2305.10431/) (2023)
- [Diffusion Model-Based Image Editing: A Survey](https://mlsanity.com/paper/2402.17525/) (2024)
- [Expressive Text-to-Image Generation with Rich Text](https://mlsanity.com/paper/2304.06720/) (2023)
- [Magic Fixup: Streamlining Photo Editing by Watching Dynamic Videos](https://mlsanity.com/paper/2403.13044/) (2024)
- [MULAN: A Multi Layer Annotated Dataset for Controllable Text-to-Image Generation](https://mlsanity.com/paper/2404.02790/) (2024)
- [ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generation](https://mlsanity.com/paper/2502.18364/) (2025)
- [Divide, Evaluate, and Refine: Evaluating and Improving Text-to-Image Alignment with Iterative VQA Feedback](https://mlsanity.com/paper/2307.04749/) (2023)
- [NoiseCollage: A Layout-Aware Text-to-Image Diffusion Model Based on Noise Cropping and Merging](https://mlsanity.com/paper/2403.03485/) (2024)
- [High-fidelity Person-centric Subject-to-Image Synthesis](https://mlsanity.com/paper/2311.10329/) (2023)
- [Alfie: Democratising RGBA Image Generation With No ](https://mlsanity.com/paper/2408.14826/) (2024)
- [Magic Insert: Style-Aware Drag-and-Drop](https://mlsanity.com/paper/2407.02489/) (2024)
- [Local Conditional Controlling for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2312.08768/) (2023)
- [Cocktail: Mixing Multi-Modality Controls for Text-Conditional Image Generation](https://mlsanity.com/paper/2306.00964/) (2023)
- [Move Anything with Layered Scene Diffusion](https://mlsanity.com/paper/2404.07178/) (2024)
- [Detector Guidance for Multi-Object Text-to-Image Generation](https://mlsanity.com/paper/2306.02236/) (2023)
- [ORIDa: Object-centric Real-world Image Composition Dataset](https://mlsanity.com/paper/2506.08964/) (2025)
- [VSC: Visual Search Compositional Text-to-Image Diffusion Model](https://mlsanity.com/paper/2505.01104/) (2025)
- [Refine-by-Align: Reference-Guided Artifacts Refinement through Semantic Alignment](https://mlsanity.com/paper/2412.00306/) (2024)
- [Diffusion Model Compression for Image-to-Image Translation](https://mlsanity.com/paper/2401.17547/) (2024)
- [BIFR"OST: 3D-Aware Image compositing with Language Instructions](https://mlsanity.com/paper/2410.19079/) (2024)
- [Text-guided Image-and-Shape Editing and Generation: A Short Survey](https://mlsanity.com/paper/2304.09244/) (2023)
- [LogoSticker: Inserting Logos into Diffusion Models for Customized Generation](https://mlsanity.com/paper/2407.13752/) (2024)
- [Generative Photomontage](https://mlsanity.com/paper/2408.07116/) (2024)
- [Zero-Shot Subject-Centric Generation for Creative Application Using Entropy Fusion](https://mlsanity.com/paper/2503.10697/) (2025)
- [Referring Layer Decomposition](https://mlsanity.com/paper/2602.19358/) (2026)
- [Streamlining Image Editing with Layered Diffusion Brushes](https://mlsanity.com/paper/2405.00313/) (2024)
- [FashionComposer: Compositional Fashion Image Generation](https://mlsanity.com/paper/2412.14168/) (2024)
- [PairHuman: A High-Fidelity Photographic Dataset for Customized Dual-Person Generation](https://mlsanity.com/paper/2511.16712/) (2025)
- [A Training-Free Approach for Multi-ID Customization via Attention Adjustment and Spatial Control](https://mlsanity.com/paper/2511.20401/) (2025)

## Abstract (arXiv)

We seek to give users precise control over diffusion-based image generation by modeling complex scenes as sequences of layers, which define the desired spatial arrangement and visual attributes of objects in the scene. Collage Diffusion harmonizes the input layers to make objects fit together -- the key challenge involves minimizing changes in the positions and key visual attributes of the input layers while allowing other attributes to change in the harmonization process. We ensure that objects are generated in the correct locations by modifying text-image cross-attention with the layers' alpha masks. We preserve key visual attributes of input layers by learning specialized text representations per layer and by extending ControlNet to operate on layers. Layer input allows users to control the extent of image harmonization on a per-object basis, and users can even iteratively edit individual objects in generated images while keeping other objects fixed. By leveraging the rich information present in layer input, Collage Diffusion generates globally harmonized images that maintain desired object characteristics better than prior approaches.