# BrushEdit: All-In-One Image Inpainting and Editing

arXiv: https://arxiv.org/abs/2412.10316 · HTML: https://mlsanity.com/paper/2412.10316/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 82 of its 103 references that are in the map.
- First posted on arXiv 13 Dec 2024.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 82)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation](https://mlsanity.com/paper/2211.12572/) (2022)
- [MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing](https://mlsanity.com/paper/2304.08465/) (2023)
- [DiffEdit: Diffusion-based semantic image editing with mask guidance](https://mlsanity.com/paper/2210.11427/) (2022)
- [DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation](https://mlsanity.com/paper/2110.02711/) (2021)

## Built on by (most cited first, 30 of 31)

- [Step1X-Edit: A Practical Framework for General Image Editing](https://mlsanity.com/paper/2504.17761/) (2025)
- [VideoPainter: Any-length Video Inpainting and Editing with Plug-and-Play Context Control](https://mlsanity.com/paper/2503.05639/) (2025)
- [KV-Edit: Training-Free Image Editing for Precise Background Preservation](https://mlsanity.com/paper/2502.17363/) (2025)
- [OmniPaint: Mastering Object-Oriented Editing via Disentangled Insertion-Removal Inpainting](https://mlsanity.com/paper/2503.08677/) (2025)
- [FireEdit: Fine-grained Instruction-based Image Editing via Region-aware Vision Language Model](https://mlsanity.com/paper/2503.19839/) (2025)
- [Does FLUX Already Know How to Perform Physically Plausible Image Composition?](https://mlsanity.com/paper/2509.21278/) (2025)
- [ReasonEdit: Towards Reasoning-Enhanced Image Editing Models](https://mlsanity.com/paper/2511.22625/) (2025)
- [Query-Kontext: An Unified Multimodal Model for Image Generation and Editing](https://mlsanity.com/paper/2509.26641/) (2025)
- [GEditBench v2: A Human-Aligned Benchmark for General Image Editing](https://mlsanity.com/paper/2603.28547/) (2026)
- [UnicEdit-10M: A Dataset and Benchmark Breaking the Scale-Quality Barrier via Unified Verification for Reasoning-Enriched Edits](https://mlsanity.com/paper/2512.02790/) (2025)
- [Reasoning to Edit: Hypothetical Instruction-Based Image Editing with Visual Reasoning](https://mlsanity.com/paper/2507.01908/) (2025)
- [From Scale to Speed: Adaptive Test-Time Scaling for Image Editing](https://mlsanity.com/paper/2603.00141/) (2026)
- [Describe, Don't Dictate: Semantic Image Editing with Natural Language Intent](https://mlsanity.com/paper/2508.20505/) (2025)
- [EditInfinity: Image Editing with Binary-Quantized Generative Models](https://mlsanity.com/paper/2510.20217/) (2025)
- [RegionE: Adaptive Region-Aware Generation for Efficient Image Editing](https://mlsanity.com/paper/2510.25590/) (2025)
- [ProEdit: Inversion-based Editing From Prompts Done Right](https://mlsanity.com/paper/2512.22118/) (2025)
- [HP-Edit: A Human-Preference Post-Training Framework for Image Editing](https://mlsanity.com/paper/2604.19406/) (2026)
- [Trans-Adapter: A Plug-and-Play Framework for Transparent Image Inpainting](https://mlsanity.com/paper/2508.01098/) (2025)
- [One Stone with Two Birds: A Null-Text-Null Frequency-Aware Diffusion Models for Text-Guided Image Inpainting](https://mlsanity.com/paper/2510.08273/) (2025)
- [MIND-Edit: MLLM Insight-Driven Editing via Language-Vision Projection](https://mlsanity.com/paper/2505.19149/) (2025)
- [Towards Reliable Identification of Diffusion-based Image Manipulations](https://mlsanity.com/paper/2506.05466/) (2025)
- [LORE: Latent Optimization for Precise Semantic Control in Rectified Flow-based Image Editing](https://mlsanity.com/paper/2508.03144/) (2025)
- [BlobCtrl: Taming Controllable Blob for Element-level Image Editing](https://mlsanity.com/paper/2503.13434/) (2025)
- [CannyEdit: Selective Canny Control and Dual-Prompt Guidance for Training-Free Image Editing](https://mlsanity.com/paper/2508.06937/) (2025)
- [CogniEdit: Dense Gradient Flow Optimization for Fine-Grained Image Editing](https://mlsanity.com/paper/2512.13276/) (2025)
- [Understanding the Implicit User Intention via Reasoning with Large Language Model for Image Editing](https://mlsanity.com/paper/2510.27335/) (2025)
- [InterCoG: Towards Spatially Precise Image Editing with Interleaved Chain-of-Grounding Reasoning](https://mlsanity.com/paper/2603.01586/) (2026)
- [ProductConsistency: Improving Product Identity Preservation in Instruction-Based Image Editing via SFT and RL](https://mlsanity.com/paper/2606.19103/) (2026)
- [UniGen-AR: Unifying Visual Generation with Auto-Regressive Modeling](https://mlsanity.com/paper/2607.24157/) (2026)
- [Instruction-based Image Editing: A Survey on Data, Models, Evaluation, and Applications](https://mlsanity.com/paper/2607.25642/) (2026)

## Abstract (arXiv)

Image editing has advanced significantly with the development of diffusion models using both inversion-based and instruction-based methods. However, current inversion-based approaches struggle with big modifications (e.g., adding or removing objects) due to the structured nature of inversion noise, which hinders substantial changes. Meanwhile, instruction-based methods often constrain users to black-box operations, limiting direct interaction for specifying editing regions and intensity. To address these limitations, we propose BrushEdit, a novel inpainting-based instruction-guided image editing paradigm, which leverages multimodal large language models (MLLMs) and image inpainting models to enable autonomous, user-friendly, and interactive free-form instruction editing. Specifically, we devise a system enabling free-form instruction editing by integrating MLLMs and a dual-branch image inpainting model in an agent-cooperative framework to perform editing category classification, main object identification, mask acquisition, and editing area inpainting. Extensive experiments show that our framework effectively combines MLLMs and inpainting models, achieving superior performance across seven metrics including mask region preservation and editing effect coherence.