mlsanityData to 13 Oct 2026

BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusion

arXiv 2403.06976 ↗First posted 11 Mar 2024cs.CVcited by 200

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Image inpainting, the process of restoring corrupted images, has seen significant advancements with the advent of diffusion models (DMs). Despite these advancements, current DM adaptations for inpainting, which involve modifications to the sampling strategy or the development of inpainting-specific DMs, frequently suffer from semantic inconsistencies and reduced image quality. Addressing these challenges, our work introduces a novel paradigm: the division of masked image features and noisy latent into separate branches. This division dramatically diminishes the model's learning load, facilitating a nuanced incorporation of essential masked image information in a hierarchical fashion. Herein, we present BrushNet, a novel plug-and-play dual-branch model engineered to embed pixel-level masked image features into any pre-trained DM, guaranteeing coherent and enhanced image inpainting outcomes. Additionally, we introduce BrushData and BrushBench to facilitate segmentation-based inpainting training and performance assessment. Our extensive experimental analysis demonstrates BrushNet's superior performance over existing models across seven key metrics, including image quality, mask region preservation, and textual coherence.

Key facts. Cited by 200 papers in the map (data through 13 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 38 of its 51 references that are in the map. First posted on arXiv 11 Mar 2024.

Builds on

38 of 51 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image Inpainting with Diffusion Models21 Dec
  2. 2023DreamInpainter: Text-Guided Subject-Driven Image Inpainting with Diffusion Models5 Dec
  3. 2023Image Inpainting via Tractable Steering of Diffusion Models28 Nov
  4. 2023Uni-paint: A Unified Framework for Multimodal Image Inpainting with Pretrained Diffusion Model11 Oct
  5. 2023Magicremover: Tuning-free Text-guided Image inpainting with Diffusion Models4 Oct
  6. 2023Towards Coherent Image Inpainting Using Denoising Diffusion Implicit Models6 Apr
  7. 2022SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Model9 Dec
  8. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Models24 Jan

Other references · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,952
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,777
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,550
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,082
  6. 2020Denoising Diffusion Implicit Modelscited by 9,718
  7. 2022Classifier-Free Diffusion Guidancecited by 5,785
  8. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,484
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,141
  10. 2014Deep Learning Face Attributes in the Wildcited by 3,686
  11. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,350
  12. 2018Demystifying MMD GANscited by 1,277
18 more
  1. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 926
  2. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 919
  3. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 871
  4. 2022Blended Latent Diffusioncited by 432
  5. 2023Rethinking FID: Towards a Better Evaluation Metric for Image Generationcited by 305
  6. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 255
  7. 2023Inpaint Anything: Segment Anything Meets Image Inpaintingcited by 233
  8. 2022Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpaintingcited by 231
  9. 2021Large Scale Image Completion via Co-Modulated Generative Adversarial Networkscited by 220
  10. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  11. 2023A Task is Worth One Word: Learning with Task Prompts for High-Quality Versatile Image Inpaintingcited by 168
  12. 2019Pluralistic Image Completioncited by 162
  13. 2023Kandinsky: an Improved Text-to-Image Synthesis with Image Prior and Latent Diffusioncited by 114
  14. 2021Generating Diverse Structure for Image Inpainting With Hierarchical VQ-VAEcited by 106
  15. 2018IntroVAE: Introspective Variational Autoencoders for Photographic Image Synthesiscited by 85
  16. 2021PD-GAN: Probabilistic Diverse GAN for Image Inpaintingcited by 80
  17. 2022CM-GAN: Image Inpainting with Cascaded Modulation GAN and Object-Aware Trainingcited by 49
  18. 2024Deep Learning-based Image and Video Inpainting: A Surveycited by 44

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 34; 2025: 96; 2026: 70.

200 papers.

Sort:

By area:

  1. 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 424
  2. 2024UltraEdit: Instruction-based Fine-Grained Image Editing at Scalecited by 267
  3. 2024OmniEdit: Building Image Editing Generalist Models Through Specialist Supervisioncited by 145
  4. 2024UniReal: Universal Image Generation and Editing via Learning Real-world Dynamicscited by 101
  5. 2024CatVTON: Concatenation Is All You Need for Virtual Try-On with Diffusion Modelscited by 100
  6. 2025ACE++: Instruction-Based Image Creation and Editing via Context-Aware Content Fillingcited by 95
  7. 2025VideoPainter: Any-length Video Inpainting and Editing with Plug-and-Play Context Controlcited by 83
  8. 2025Se~norita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialistscited by 61
  9. 2025KV-Edit: Training-Free Image Editing for Precise Background Preservationcited by 48
  10. 2024Region-Aware Text-to-Image Generation via Hard Binding and Soft Refinementcited by 37
  11. 2025POSTA: A Go-to Framework for Customized Artistic Poster Generationcited by 36
  12. 2024FreeEdit: Mask-free Reference-based Image Editing with Multi-modal Instructioncited by 35
38 more
  1. 2025OmniEraser: Remove Objects and Their Effects in Images with Paired Video-Frame Datacited by 35
  2. 2025ROSE: Remove Objects with Side Effects in Videoscited by 34
  3. 2024Generative Image Layer Decomposition with Visual Effectscited by 33
  4. 2024BrushEdit: All-In-One Image Inpainting and Editingcited by 31
  5. 2024MagicQuill: An Intelligent Interactive Image Editing Systemcited by 30
  6. 2025Does FLUX Already Know How to Perform Physically Plausible Image Composition?cited by 30
  7. 2025NANO3D: A Training-Free Approach for Efficient 3D Editing Without Maskscited by 29
  8. 2024DepthLab: From Partial to Completecited by 27
  9. 2025UniCombine: Unified Multi-Conditional Combination with Diffusion Transformercited by 25
  10. 2024VIVID-10M: A Dataset and Baseline for Versatile and Interactive Video Local Editingcited by 24
  11. 2025SmartEraser: Remove Anything from Images using Masked-Region Guidancecited by 24
  12. 2025StyleSSP: Sampling StartPoint Enhancement for Training-free Diffusion-based Method for Style Transfercited by 23
  13. 2024SharpDepth: Sharpening Metric Depth Predictions Using Diffusion Distillationcited by 21
  14. 2025UltraFusion: Ultra High Dynamic Imaging using Exposure Fusioncited by 20
  15. 2025HiScene: Creating Hierarchical 3D Scenes with Isometric View Generationcited by 18
  16. 2025Follow-Your-Preference: Towards Preference-Aligned Image Inpaintingcited by 18
  17. 2024PrefPaint: Aligning Image Inpainting Diffusion Model with Human Preferencecited by 17
  18. 2024InsightEdit: Towards Better Instruction Following for Image Editingcited by 17
  19. 2024Reflecting Reality: Enabling Diffusion Models to Produce Faithful Mirror Reflectionscited by 16
  20. 2025Zooming In on Fakes: A Novel Dataset for Localized AI-Generated Image Detection with Forgery Amplification Approachcited by 16
  21. 2025SceneMaker: Open-set 3D Scene Generation with Decoupled De-occlusion and Pose Estimation Modelcited by 16
  22. 2025TurboFill: Adapting Few-step Text-to-image Model for Fast Image Inpaintingcited by 15
  23. 2025MTV-Inpaint: Multi-Task Long Video Inpaintingcited by 14
  24. 2024UniPaint: Unified Space-time Video Inpainting via Mixture-of-Expertscited by 13
  25. 2025Precise Object and Effect Removal with Adaptive Target-Aware Attentioncited by 13
  26. 2025Training-free Geometric Image Editing on Diffusion Modelscited by 13
  27. 2025Flux Already Knows -- Activating Subject-Driven Image Generation without Trainingcited by 11
  28. 2026Group Editing: Edit Multiple Images in One Gocited by 11
  29. 2025SAGI: Semantically Aligned and Uncertainty Guided AI Image Inpaintingcited by 10
  30. 2025TACO: Taming Diffusion for in-the-wild Video Amodal Completioncited by 10
  31. 2025Frame In-N-Out: Unbounded Controllable Image-to-Video Generationcited by 10
  32. 2024Edicho: Consistent Image Editing in the Wildcited by 9
  33. 2025OmniVTON: Training-Free Universal Virtual Try-Oncited by 9
  34. 2025GeoRemover: Removing Objects and Their Causal Visual Artifactscited by 9
  35. 2025MirrorVerse: Pushing Diffusion Models to Realistically Reflect the Worldcited by 8
  36. 2025DreamLight: Towards Harmonious and Consistent Image Relightingcited by 8
  37. 2025MIRAGE: Towards AI-Generated Image Detection in the Wildcited by 8
  38. 2026PosterOmni: Generalized Artistic Poster Creation via Task Distillation and Unified Reward Feedbackcited by 8

50 most cited of 200 All 200 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025GuidPaint: Class-Guided Image Inpainting with Diffusion Models
  2. 2024Paint by Inpaint: Learning to Add Image Objects by Removing Them First
  3. 2024Diffusion-based image inpainting with internal learning
  4. 2021Learning Sparse Masks for Diffusion-based Image Inpainting
  5. 2023A Theoretical Justification for Image Inpainting using Denoising Diffusion Probabilistic Models
  6. 2024RAD: Region-Aware Diffusion Models for Image Inpainting
  7. 2023DiffGANPaint: Fast Inpainting Using Denoising Diffusion GANs
  8. 20243D-Consistent Image Inpainting with Diffusion Models