# Iterative Multi-granular Image Editing using Diffusion Models

arXiv: https://arxiv.org/abs/2309.00613 · HTML: https://mlsanity.com/paper/2309.00613/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 31 of its 37 references that are in the map.
- First posted on arXiv 1 Sep 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 31)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation](https://mlsanity.com/paper/2211.12572/) (2022)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)
- [DiffEdit: Diffusion-based semantic image editing with mask guidance](https://mlsanity.com/paper/2210.11427/) (2022)
- [Compositional Visual Generation with Composable Diffusion Models](https://mlsanity.com/paper/2206.01714/) (2022)
- [StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1612.03242/) (2016)
- [Blended Latent Diffusion](https://mlsanity.com/paper/2206.02779/) (2022)
- [Pix2Video: Video Editing using Image Diffusion](https://mlsanity.com/paper/2303.12688/) (2023)
- [Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpainting](https://mlsanity.com/paper/2212.06909/) (2022)
- [Unifying Diffusion Models' Latent Space, with Applications to CycleDiffusion and Guidance](https://mlsanity.com/paper/2210.05559/) (2022)
- [Photographic Text-to-Image Synthesis with a Hierarchically-nested Adversarial Network](https://mlsanity.com/paper/1802.09178/) (2018)
- [SSCR: Iterative Language-Based Image Editing via Self-Supervised Counterfactual Reasoning](https://mlsanity.com/paper/2009.09566/) (2020)
- [OPA: Object Placement Assessment Dataset](https://mlsanity.com/paper/2107.01889/) (2021)
- [Learning Object Placement via Dual-path Graph Completion](https://mlsanity.com/paper/2207.11464/) (2022)

## Built on by (most cited first, 20 of 20)

- [Diffusion Model-Based Image Editing: A Survey](https://mlsanity.com/paper/2402.17525/) (2024)
- [Direct Inversion: Boosting Diffusion-based Editing with 3 Lines of Code](https://mlsanity.com/paper/2310.01506/) (2023)
- [Focus on Your Instruction: Fine-grained and Multi-instruction Image Editing by Attention Modulation](https://mlsanity.com/paper/2312.10113/) (2023)
- [BrushEdit: All-In-One Image Inpainting and Editing](https://mlsanity.com/paper/2412.10316/) (2024)
- [LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction Tuning](https://mlsanity.com/paper/2312.03849/) (2023)
- [Multi-turn Consistent Image Editing](https://mlsanity.com/paper/2505.04320/) (2025)
- [ByteEdit: Boost, Comply and Accelerate Generative Image Editing](https://mlsanity.com/paper/2404.04860/) (2024)
- [LayerEdit: Disentangled Multi-Object Editing via Conflict-Aware Multi-Layer Learning](https://mlsanity.com/paper/2511.08251/) (2025)
- [IMAGAgent: Orchestrating Multi-Turn Image Editing via Constraint-Aware Planning and Reflection](https://mlsanity.com/paper/2603.29602/) (2026)
- [ParallelEdits: Efficient Multi-object Image Editing](https://mlsanity.com/paper/2406.00985/) (2024)
- [REED-VAE: RE-Encode Decode Training for Iterative Image Editing with Diffusion Models](https://mlsanity.com/paper/2504.18989/) (2025)
- [Is This Edit Correct? A Multi-Dimensional Benchmark for Reasoning-Aware Image Editing](https://mlsanity.com/paper/2606.05172/) (2026)
- [Streamlining Image Editing with Layered Diffusion Brushes](https://mlsanity.com/paper/2405.00313/) (2024)
- [Specify and Edit: Overcoming Ambiguity in Text-Based Image Editing](https://mlsanity.com/paper/2407.20232/) (2024)
- [CAMILA: Context-Aware Masking for Image Editing with Language Alignment](https://mlsanity.com/paper/2509.19731/) (2025)
- [Unsupervised Panoptic Interpretation of Latent Spaces in GANs Using Space-Filling Vector Quantization](https://mlsanity.com/paper/2410.20573/) (2024)
- [VIIS: Visible and Infrared Information Synthesis for Severe Low-light Image Enhancement](https://mlsanity.com/paper/2412.13655/) (2024)
- [Disentangling Instruction Influence in Diffusion Transformers for Parallel Multi-Instruction-Guided Image Editing](https://mlsanity.com/paper/2504.04784/) (2025)
- [RemEdit: Efficient Diffusion Editing with Riemannian Geometry](https://mlsanity.com/paper/2601.17927/) (2026)
- [RegionCache: Semantic-Aware Region Reuse for Efficient Multi-Turn Image Generation](https://mlsanity.com/paper/2608.29809/) (2026)

## Abstract (arXiv)

Recent advances in text-guided image synthesis has dramatically changed how creative professionals generate artistic and aesthetically pleasing visual assets. To fully support such creative endeavors, the process should possess the ability to: 1) iteratively edit the generations and 2) control the spatial reach of desired changes (global, local or anything in between). We formalize this pragmatic problem setting as Iterative Multi-granular Editing. While there has been substantial progress with diffusion-based models for image synthesis and editing, they are all one shot (i.e., no iterative editing capabilities) and do not naturally yield multi-granular control (i.e., covering the full spectrum of local-to-global edits). To overcome these drawbacks, we propose EMILIE: Iterative Multi-granular Image Editor. EMILIE introduces a novel latent iteration strategy, which re-purposes a pre-trained diffusion model to facilitate iterative editing. This is complemented by a gradient control operation for multi-granular control. We introduce a new benchmark dataset to evaluate our newly proposed setting. We conduct exhaustive quantitatively and qualitatively evaluation against recent state-of-the-art approaches adapted to our task, to being out the mettle of EMILIE. We hope our work would attract attention to this newly identified, pragmatic problem setting.