# DesignEdit: Multi-Layered Latent Decomposition and Fusion for Unified & Accurate Image Editing

arXiv: https://arxiv.org/abs/2403.14487 · HTML: https://mlsanity.com/paper/2403.14487/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 36 references that are in the map.
- First posted on arXiv 21 Mar 2024.

## Builds on (most cited first, 30 of 32)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [Vector Quantized Diffusion Model for Text-to-Image Synthesis](https://mlsanity.com/paper/2111.14822/) (2021)
- [Resolution-robust Large Mask Inpainting with Fourier Convolutions](https://mlsanity.com/paper/2109.07161/) (2021)
- [MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing](https://mlsanity.com/paper/2304.08465/) (2023)
- [Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2301.13826/) (2023)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing](https://mlsanity.com/paper/2306.10012/) (2023)
- [Blended Latent Diffusion](https://mlsanity.com/paper/2206.02779/) (2022)
- [Training-Free Layout Control with Cross-Attention Guidance](https://mlsanity.com/paper/2304.03373/) (2023)
- [Diffusion Self-Guidance for Controllable Image Generation](https://mlsanity.com/paper/2306.00986/) (2023)
- [FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model](https://mlsanity.com/paper/2303.09833/) (2023)
- [DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing](https://mlsanity.com/paper/2306.14435/) (2023)
- [BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion](https://mlsanity.com/paper/2307.10816/) (2023)
- [Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold](https://mlsanity.com/paper/2305.10973/) (2023)
- [Style Aligned Image Generation via Shared Attention](https://mlsanity.com/paper/2312.02133/) (2023)
- [Subject-driven Text-to-Image Generation via Apprenticeship Learning](https://mlsanity.com/paper/2304.00186/) (2023)
- [DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Models](https://mlsanity.com/paper/2307.02421/) (2023)
- [DiffEditor: Boosting Accuracy and Flexibility on Diffusion-based Image Editing](https://mlsanity.com/paper/2402.02583/) (2024)

## Built on by (most cited first, 20 of 20)

- [Insert Anything: Image Insertion via In-Context Editing in DiT](https://mlsanity.com/paper/2504.15009/) (2025)
- [FreeEdit: Mask-free Reference-based Image Editing with Multi-modal Instruction](https://mlsanity.com/paper/2409.18071/) (2024)
- [Generative Image Layer Decomposition with Visual Effects](https://mlsanity.com/paper/2411.17864/) (2024)
- [RORem: Training a Robust Object Remover with Human-in-the-Loop](https://mlsanity.com/paper/2501.00740/) (2025)
- [DreamLayer: Simultaneous Multi-Layer Generation via Diffusion Mode](https://mlsanity.com/paper/2503.12838/) (2025)
- [Precise Object and Effect Removal with Adaptive Target-Aware Attention](https://mlsanity.com/paper/2505.22636/) (2025)
- [Training-free Geometric Image Editing on Diffusion Models](https://mlsanity.com/paper/2507.23300/) (2025)
- [Rethinking Layered Graphic Design Generation with a Top-Down Approach](https://mlsanity.com/paper/2507.05601/) (2025)
- [Consistent Image Layout Editing with Diffusion Models](https://mlsanity.com/paper/2503.06419/) (2025)
- [LayerEdit: Disentangled Multi-Object Editing via Conflict-Aware Multi-Layer Learning](https://mlsanity.com/paper/2511.08251/) (2025)
- [AIComposer: Any Style and Content Image Composition via Feature Integration](https://mlsanity.com/paper/2507.20721/) (2025)
- [OmniText: A Training-Free Generalist for Controllable Text-Image Manipulation](https://mlsanity.com/paper/2510.24093/) (2025)
- [Free-Form Scene Editor: Enabling Multi-Round Object Manipulation like in a 3D Engine](https://mlsanity.com/paper/2511.13713/) (2025)
- [Geometric Image Editing via Effects-Sensitive In-Context Inpainting with Diffusion Transformers](https://mlsanity.com/paper/2602.08388/) (2026)
- [FlashClear: Ultra-Fast Image Content Removal via Efficient Step Distillation and Feature Caching](https://mlsanity.com/paper/2605.09003/) (2026)
- [Does Synthetic Layered Design Data Benefit Layered Design Decomposition?](https://mlsanity.com/paper/2605.15167/) (2026)
- [MILD: Multi-Layer Diffusion Strategy for Complex and Precise Multi-IP Aware Human Erasing](https://mlsanity.com/paper/2508.06543/) (2025)
- [AdaEraser: Training-Free Object Removal via Adaptive Attention Suppression](https://mlsanity.com/paper/2605.15921/) (2026)
- [TurboClear: One-Step Object-Effect Removal via Region-Calibrated Distribution Matching and Fusion](https://mlsanity.com/paper/2608.01288/) (2026)
- [QK-Edit: Revisiting Attention-based Injection in MM-DiT for Image and Video Editing](https://mlsanity.com/paper/2610.17829/) (2026)

## Abstract (arXiv)

Recently, how to achieve precise image editing has attracted increasing attention, especially given the remarkable success of text-to-image generation models. To unify various spatial-aware image editing abilities into one framework, we adopt the concept of layers from the design domain to manipulate objects flexibly with various operations. The key insight is to transform the spatial-aware image editing task into a combination of two sub-tasks: multi-layered latent decomposition and multi-layered latent fusion. First, we segment the latent representations of the source images into multiple layers, which include several object layers and one incomplete background layer that necessitates reliable inpainting. To avoid extra tuning, we further explore the inner inpainting ability within the self-attention mechanism. We introduce a key-masking self-attention scheme that can propagate the surrounding context information into the masked region while mitigating its impact on the regions outside the mask. Second, we propose an instruction-guided latent fusion that pastes the multi-layered latent representations onto a canvas latent. We also introduce an artifact suppression scheme in the latent space to enhance the inpainting quality. Due to the inherent modular advantages of such multi-layered representations, we can achieve accurate image editing, and we demonstrate that our approach consistently surpasses the latest spatial editing methods, including Self-Guidance and DiffEditor. Last, we show that our approach is a unified framework that supports various accurate image editing tasks on more than six different editing tasks.