mlsanityData to 7 Oct 2026

ObjectDrop: Bootstrapping Counterfactuals for Photorealistic Object Removal and Insertion

arXiv 2403.18818 ↗First posted 27 Mar 2024cs.CVcited by 71

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Diffusion models have revolutionized image editing but often generate images that violate physical laws, particularly the effects of objects on the scene, e.g., occlusions, shadows, and reflections. By analyzing the limitations of self-supervised approaches, we propose a practical solution centered on a qcounterfactual dataset. Our method involves capturing a scene before and after removing a single object, while minimizing other changes. By fine-tuning a diffusion model on this dataset, we are able to not only remove objects but also their effects on the scene. However, we find that applying this approach for photorealistic object insertion requires an impractically large dataset. To tackle this challenge, we propose bootstrap supervision; leveraging our object removal model trained on a small counterfactual dataset, we synthetically expand this dataset considerably. Our approach significantly outperforms prior methods in photorealistic object removal and insertion, particularly at modeling the effects of objects on the scene.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 53 of its 64 references that are in the map. First posted on arXiv 27 Mar 2024.

Builds on

53 of 64 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Latent Feature-Guided Diffusion Models for Shadow Removal4 Dec
  2. 2022ShadowDiffusion: When Degradation Prior Meets Diffusion Model for Shadow Removal9 Dec
  3. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Models24 Jan

Other references · 50

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  6. 2023Segment Anythingcited by 8,158
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  12. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
38 more
  1. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  2. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  3. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  4. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  5. 2019StarGAN v2: Diverse Image Synthesis for Multiple Domainscited by 1,095
  6. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 929
  7. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  8. 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 730
  9. 2018Image Inpainting for Irregular Holes Using Partial Convolutionscited by 505
  10. 2019Variational Autoencoders and Nonlinear ICA: A Unifying Frameworkcited by 494
  11. 2022Paint by Example: Exemplar-based Image Editing with Diffusion Modelscited by 489
  12. 2023AnyDoor: Zero-shot Object-level Image Customizationcited by 411
  13. 2023Emu Edit: Precise Image Editing via Recognition and Generation Taskscited by 321
  14. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  15. 2022Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpaintingcited by 229
  16. 2023Guiding Instruction-based Image Editing via Multimodal Large Language Modelscited by 182
  17. 2023HIVE: Harnessing Human Feedback for Instructional Visual Editingcited by 172
  18. 2017Stacked Conditional Generative Adversarial Networks for Jointly Learning Shadow Detection and Shadow Removalcited by 160
  19. 2019StructureFlow: Image Inpainting via Structure-aware Appearance Flowcited by 118
  20. 2019Towards Ghost-free Shadow Removal via Dual Hierarchical Aggregation Network and Shadow Matting GANcited by 102
  21. 2020Rethinking Image Inpainting via a Mutual Encoder-Decoder with Feature Equalizationscited by 102
  22. 2019Shadow Removal via Shadow Image Decompositioncited by 96
  23. 2019Learning Pyramid-Context Encoder Network for High-Quality Image Inpaintingcited by 94
  24. 2019Mask-ShadowGAN: Learning to Remove Shadows from Unpaired Datacited by 91
  25. 2021Auto-Exposure Fusion for Single-Image Shadow Removalcited by 78
  26. 2022NUWA-Infinity: Autoregressive over Autoregressive Generation for Infinite Visual Synthesiscited by 71
  27. 2022DC-ShadowNet: Single-Image Hard and Soft Shadow Removal Using Unsupervised Domain-Classifier Guided Networkcited by 68
  28. 2020From Shadow Segmentation to Shadow Removalcited by 59
  29. 2021From Shadow Generation to Shadow Removalcited by 54
  30. 2023Putting People in Their Place: Affordance-Aware Human Insertion into Scenescited by 53
  31. 2019ARGAN: Attentive Recurrent Generative Adversarial Network for Shadow Detection and Removalcited by 51
  32. 2021Omnimatte: Associating Objects and Their Effects in Videocited by 50
  33. 2019Instance Shadow Detectioncited by 37
  34. 2021Shadow Generation for Composite Image in Real-world Scenescited by 35
  35. 2021An Image is Worth More Than a Thousand Words: Towards Disentanglement in the Wildcited by 30
  36. 2020Image Fine-grained Inpaintingcited by 16
  37. 2022A Contrastive Objective for Learning Disentangled Representationscited by 14
  38. 2020AIM 2020 Challenge on Image Extreme Inpaintingcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 35; 2026: 24.

71 papers.

Sort:

By area:

  1. 2024Generative Omnimatte: Learning to Decompose Video into Layerscited by 38
  2. 2025OmniPaint: Mastering Object-Oriented Editing via Disentangled Insertion-Removal Inpaintingcited by 38
  3. 2025OmniEraser: Remove Objects and Their Effects in Images with Paired Video-Frame Datacited by 35
  4. 2024Generative Image Layer Decomposition with Visual Effectscited by 33
  5. 2025NoHumansRequired: Autonomous High-Quality Image Editing Triplet Miningcited by 33
  6. 2025ObjectMover: Generative Object Movement with Video Priorcited by 30
  7. 2025SmartEraser: Remove Anything from Images using Masked-Region Guidancecited by 24
  8. 2025RORem: Training a Robust Object Remover with Human-in-the-Loopcited by 23
  9. 2024DiffUHaul: A Training-Free Method for Object Dragging in Imagescited by 22
  10. 2024CLIPAway: Harmonizing Focused Embeddings for Removing Objects via Diffusion Modelscited by 19
  11. 2025PlanGen: Towards Unified Layout Planning and Image Generation in Auto-Regressive Vision Language Modelscited by 18
  12. 2024Magic Insert: Style-Aware Drag-and-Dropcited by 17
38 more
  1. 2025Generative AI for Cel-Animation: A Surveycited by 17
  2. 2025IntrinsicEdit: Precise generative image manipulation in intrinsic spacecited by 17
  3. 2025R3D2: Realistic 3D Asset Insertion via Diffusion for Autonomous Driving Simulationcited by 17
  4. 2024Reflecting Reality: Enabling Diffusion Models to Produce Faithful Mirror Reflectionscited by 16
  5. 2024ObjectMate: A Recurrence Prior for Object Insertion and Subject-Driven Generationcited by 16
  6. 2025OmnimatteZero: Fast Training-free Omnimatte with Pre-trained Video Diffusion Modelscited by 15
  7. 2025VideoHandles: Editing 3D Object Compositions in Videos Using Video Generative Priorscited by 14
  8. 2025DreamFuse: Adaptive Image Fusion with Diffusion Transformercited by 13
  9. 2025Precise Object and Effect Removal with Adaptive Target-Aware Attentioncited by 13
  10. 2025Learning an Image Editing Model without Image Editing Pairscited by 13
  11. 20253D-Fixup: Advancing Photo Editing with 3D Priorscited by 12
  12. 2026EffectErase: Joint Video Object Removal and Insertion for High-Quality Effect Erasingcited by 12
  13. 2025Visual Jenga: Discovering Object Dependencies via Counterfactual Inpaintingcited by 11
  14. 2025ORIDa: Object-centric Real-world Image Composition Datasetcited by 11
  15. 2026Group Editing: Edit Multiple Images in One Gocited by 11
  16. 2025DynVFX: Augmenting Real Videos with Dynamic Contentcited by 9
  17. 2025GeoRemover: Removing Objects and Their Causal Visual Artifactscited by 9
  18. 2025Shape-for-Motion: Precise and Consistent Video Editing with 3D Proxycited by 8
  19. 2025In-Context Brush: Zero-shot Customized Subject Insertion with Context-Aware Latent Space Manipulationcited by 7
  20. 2025Shining Yourself: High-Fidelity Ornaments Virtual Try-on with Diffusion Modelcited by 6
  21. 2024InsertDiffusion: Identity Preserving Visualization of Objects through a Training-Free Diffusion Architecturecited by 5
  22. 2024Controllable Human Image Generation with Personalized Multi-Garmentscited by 5
  23. 2024MetaShadow: Object-Centered Shadow Detection, Removal, and Synthesiscited by 5
  24. 20253D Scene Understanding Through Local Random Access Sequence Modelingcited by 5
  25. 2024PixelMan: Consistent Object Editing with Diffusion Models via Pixel Manipulation and Generationcited by 4
  26. 2026Referring Layer Decompositioncited by 3
  27. 2024FunEditor: Achieving Complex Image Edits via Function Aggregation with Diffusion Modelscited by 2
  28. 2025Controllable 3D Placement of Objects with Scene-Aware Diffusion Modelscited by 2
  29. 2025TAUE: Training-free Noise Transplant and Cultivation Diffusion Modelcited by 2
  30. 2026VIBE: Visual Instruction Based Editorcited by 2
  31. 2025From Gallery to Wrist: Realistic 3D Bracelet Insertion in Videoscited by 1
  32. 2025CrimEdit: Controllable Editing for Counterfactual Object Removal, Insertion, and Movementcited by 1
  33. 2025Illustrator's Depth: Monocular Layer Index Prediction for Image Decompositioncited by 1
  34. 2025EasyOmnimatte: Taming Pretrained Inpainting Diffusion Models for End-to-End Video Layered Decompositioncited by 1
  35. 2026Geometric Image Editing via Effects-Sensitive In-Context Inpainting with Diffusion Transformerscited by 1
  36. 2026FlashClear: Ultra-Fast Image Content Removal via Efficient Step Distillation and Feature Cachingcited by 1
  37. 2026BlazeEdit: Generalist Image Editing on Mobile Devices with Image-to-Image Diffusion Modelscited by 1
  38. 2026DORS: Dynamic Attention Routing for Diffusion-based Object Removal in Dense Scenescited by 1

50 most cited of 71 All 71 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026VOID: Video Object and Interaction Deletion
  2. 2024ReMOVE: A Reference-free Metric for Object Erasure
  3. 2024Paint by Inpaint: Learning to Add Image Objects by Removing Them First
  4. 2024EraseDraw: Learning to Draw Step-by-Step via Erasing Objects from Images
  5. 2023Diffusion Handles: Enabling 3D Edits for Diffusion Models by Lifting Activations to 3D
  6. 2024Towards a Training Free Approach for 3D Scene Editing
  7. 2024LoMOE: Localized Multi-Object Editing via Multi-Diffusion
  8. 2026ClickRemoval: An Interactive Open-Source Tool for Object Removal in Diffusion Models