mlsanityData to 7 Oct 2026

Inst-Inpaint: Instructing to Remove Objects with Diffusion Models

arXiv 2304.03246 ↗First posted 6 Apr 2023cs.CVcited by 66

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Image inpainting task refers to erasing unwanted pixels from images and filling them in a semantically consistent and realistic way. Traditionally, the pixels that are wished to be erased are defined with binary masks. From the application point of view, a user needs to generate the masks for the objects they would like to remove which can be time-consuming and prone to errors. In this work, we are interested in an image inpainting algorithm that estimates which object to be removed based on natural language input and removes it, simultaneously. For this purpose, first, we construct a dataset named GQA-Inpaint for this task. Second, we present a novel inpainting framework, Inst-Inpaint, that can remove objects from images based on the instructions given as text prompts. We set various GAN and diffusion-based baselines and run experiments on synthetic and real image datasets. We compare methods with different evaluation metrics that measure the quality and accuracy of the models and show significant quantitative and qualitative improvements.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 43 of its 57 references that are in the map. First posted on arXiv 6 Apr 2023.

Builds on

43 of 57 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Models24 Jan

Other references · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2017Mask R-CNNcited by 7,259
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  11. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  12. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
30 more
  1. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  2. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  3. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  4. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  5. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  6. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  7. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  8. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  9. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  10. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  11. 2019Panoptic Feature Pyramid Networkscited by 644
  12. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  13. 2018Free-Form Image Inpainting with Gated Convolutioncited by 594
  14. 2021Image Segmentation Using Text and Image Promptscited by 509
  15. 2018Image Inpainting for Irregular Holes Using Partial Convolutionscited by 505
  16. 2019Cascade R-CNN: High Quality Object Detection and Instance Segmentationcited by 428
  17. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  18. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  19. 2018Composing Text and Image for Image Retrieval - An Empirical Odysseycited by 234
  20. 2021Large Scale Image Completion via Co-Modulated Generative Adversarial Networkscited by 219
  21. 2022SpaText: Spatio-Textual Representation for Controllable Image Generationcited by 217
  22. 2022MAT: Mask-Aware Transformer for Large Hole Image Inpaintingcited by 216
  23. 2020CascadePSP: Toward Class-Agnostic and Very High-Resolution Segmentation via Global and Local Refinementcited by 113
  24. 2020Recurrent Feature Reasoning for Image Inpaintingcited by 107
  25. 2018Tell, Draw, and Repeat: Generating and Modifying Images Based on Continual Linguistic Instructioncited by 61
  26. 2022High-Fidelity Guided Image Synthesis with Latent Diffusion Modelscited by 36
  27. 2022ObjectStitch: Generative Object Compositingcited by 36
  28. 2020Text as Neural Operator: Image Manipulation by Text Instructioncited by 26
  29. 2022High-Fidelity Image Inpainting with GAN Inversioncited by 15
  30. 2022Target-Free Text-guided Image Manipulationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 27; 2025: 25; 2026: 4.

66 papers.

Sort:

By area:

  1. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  2. 2023InstructDiffusion: A Generalist Modeling Interface for Vision Taskscited by 162
  3. 2024Attentive Eraser: Unleashing Diffusion Model's Object Removal Potential via Self-Attention Redirection Guidancecited by 63
  4. 2024Understanding the Impact of Negative Prompts: When and How Do They Take Effect?cited by 50
  5. 2024PromptFix: You Prompt and We Fix the Photocited by 40
  6. 2024Paint by Inpaint: Learning to Add Image Objects by Removing Them Firstcited by 39
  7. 2024Efficient Diffusion Models: A Comprehensive Survey from Principles to Practicescited by 39
  8. 2023Amodal Completion via Progressive Mixed Context Diffusioncited by 38
  9. 2025OmniPaint: Mastering Object-Oriented Editing via Disentangled Insertion-Removal Inpaintingcited by 38
  10. 2023ElasticDiffusion: Training-free Arbitrary Size Image Generation through Global-Local Content Separationcited by 37
  11. 2025OmniEraser: Remove Objects and Their Effects in Images with Paired Video-Frame Datacited by 35
  12. 2024Towards Language-Driven Video Inpainting via Multimodal Large Language Modelscited by 33
38 more
  1. 2024Learning Action and Reasoning-Centric Image Editing from Videos and Simulationscited by 32
  2. 2024PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructionscited by 31
  3. 2023DreamInpainter: Text-Guided Subject-Driven Image Inpainting with Diffusion Modelscited by 30
  4. 2024Conditional Image Synthesis with Diffusion Models: A Surveycited by 29
  5. 2025SmartEraser: Remove Anything from Images using Masked-Region Guidancecited by 24
  6. 2025RORem: Training a Robust Object Remover with Human-in-the-Loopcited by 23
  7. 2023Mini-DALLE3: Interactive Text to Image by Prompting Large Language Modelscited by 22
  8. 2025OneReward: Unified Mask-Guided Image Generation via Multi-Task Human Preference Learningcited by 22
  9. 2023Diverse Inpainting and Editing with GAN Inversioncited by 20
  10. 2024FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehensioncited by 20
  11. 2024CLIPAway: Harmonizing Focused Embeddings for Removing Objects via Diffusion Modelscited by 19
  12. 2025Erase Diffusion: Empowering Object Removal Through Calibrating Diffusion Pathwayscited by 17
  13. 2023Magicremover: Tuning-free Text-guided Image inpainting with Diffusion Modelscited by 16
  14. 2024MagicEraser: Erasing Any Objects via Semantics-Aware Controlcited by 15
  15. 2024ZIM: Zero-Shot Image Matting for Anythingcited by 15
  16. 2023InstructP2P: Learning to Edit 3D Point Clouds with Text Instructionscited by 14
  17. 2025MTV-Inpaint: Multi-Task Long Video Inpaintingcited by 14
  18. 2025Lumina-OmniLV: A Unified Multimodal Framework for General Low-Level Visioncited by 14
  19. 2025Fine-Tuning Visual Autoregressive Models for Subject-Driven Generationcited by 13
  20. 2024ReMOVE: A Reference-free Metric for Object Erasurecited by 12
  21. 2025ForgetMe: Evaluating Selective Forgetting in Generative Modelscited by 11
  22. 2024ByteEdit: Boost, Comply and Accelerate Generative Image Editingcited by 10
  23. 2025MIGE: Mutually Enhanced Multimodal Instruction-Based Image Generation and Editingcited by 10
  24. 2025Instruct-CLIP: Improving Instruction-Guided Image Editing with Automated Data Refinement Using Contrastive Learningcited by 9
  25. 2025Omni2: Unifying Omnidirectional Image Generation and Editing in an Omni Modelcited by 9
  26. 2025GeoRemover: Removing Objects and Their Causal Visual Artifactscited by 9
  27. 2025MedEBench: Diagnosing Reliability in Text-Guided Medical Image Editingcited by 8
  28. 2024InstructRL4Pix: Training Diffusion for Image Editing by Reinforcement Learningcited by 7
  29. 2025New Dataset and Methods for Fine-Grained Compositional Referring Expression Comprehension via Specialist-MLLM Collaborationcited by 7
  30. 2024EditScribe: Non-Visual Image Editing with Natural Language Verification Loopscited by 6
  31. 2023Diffusion Models for Generative Artificial Intelligence: An Introduction for Applied Mathematicianscited by 5
  32. 2024AudioScenic: Audio-Driven Video Scene Editingcited by 5
  33. 2024SOEDiff: Efficient Distillation for Small Object Editingcited by 5
  34. 2024EraseDraw: Learning to Draw Step-by-Step via Erasing Objects from Imagescited by 4
  35. 2024ColorEdit: Training-free Image-Guided Color editing with diffusion modelcited by 4
  36. 2026Instruction-based Image Editing with Planning, Reasoning, and Generationcited by 4
  37. 2025RePainter: Empowering E-commerce Object Removal via Spatial-matting Reinforcement Learningcited by 3
  38. 2026TalkPhoto: A Versatile Training-Free Conversational Assistant for Intelligent Image Editingcited by 3

50 most cited of 66 All 66 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023DiffGANPaint: Fast Inpainting Using Denoising Diffusion GANs
  2. 2025IS-Diff: Improving Diffusion-Based Inpainting with Better Initial Seed
  3. 2025GuidPaint: Class-Guided Image Inpainting with Diffusion Models
  4. 2024Diffusion-based image inpainting with internal learning
  5. 2024BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusion
  6. 2024Deep Learning-based Image and Video Inpainting: A Survey
  7. 2024BrushEdit: All-In-One Image Inpainting and Editing
  8. 2025InpDiffusion: Image Inpainting Localization via Conditional Diffusion Models