# Attentive Eraser: Unleashing Diffusion Model's Object Removal Potential via Self-Attention Redirection Guidance

arXiv: https://arxiv.org/abs/2412.12974 · HTML: https://mlsanity.com/paper/2412.12974/

## Key facts

- Cited by 63 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 46 of its 81 references that are in the map.
- First posted on arXiv 17 Dec 2024.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 46)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [DiffEdit: Diffusion-based semantic image editing with mask guidance](https://mlsanity.com/paper/2210.11427/) (2022)
- [Generative Image Inpainting with Contextual Attention](https://mlsanity.com/paper/1801.07892/) (2018)
- [Diffusion Self-Guidance for Controllable Image Generation](https://mlsanity.com/paper/2306.00986/) (2023)
- [SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Model](https://mlsanity.com/paper/2212.05034/) (2022)
- [A Task is Worth One Word: Learning with Task Prompts for High-Quality Versatile Image Inpainting](https://mlsanity.com/paper/2312.03594/) (2023)
- [InstructDiffusion: A Generalist Modeling Interface for Vision Tasks](https://mlsanity.com/paper/2309.03895/) (2023)
- [Improving Sample Quality of Diffusion Models Using Self-Attention Guidance](https://mlsanity.com/paper/2210.00939/) (2022)
- [Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing](https://mlsanity.com/paper/2403.03431/) (2024)
- [DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understanding](https://mlsanity.com/paper/2311.11810/) (2023)
- [Advancing Pose-Guided Image Synthesis with Progressive Conditional Diffusion Models](https://mlsanity.com/paper/2310.06313/) (2023)
- [FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Model with Any Condition](https://mlsanity.com/paper/2312.07536/) (2023)
- [Shift-Net: Image Inpainting via Deep Feature Rearrangement](https://mlsanity.com/paper/1801.09392/) (2018)
- [Dynamic Prompt Learning: Addressing Cross-Attention Leakage for Text-Based Image Editing](https://mlsanity.com/paper/2309.15664/) (2023)
- [TabPedia: Towards Comprehensive Visual Table Understanding with Concept Synergy](https://mlsanity.com/paper/2406.01326/) (2024)
- [SPTS v2: Single-Point Scene Text Spotting](https://mlsanity.com/paper/2301.01635/) (2023)
- [Inst-Inpaint: Instructing to Remove Objects with Diffusion Models](https://mlsanity.com/paper/2304.03246/) (2023)
- [Get What You Want, Not What You Don't: Image Content Suppression for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2402.05375/) (2024)
- [ZONE: Zero-Shot Instruction-Guided Local Editing](https://mlsanity.com/paper/2312.16794/) (2023)
- [TextSquare: Scaling up Text-Centric Visual Instruction Tuning](https://mlsanity.com/paper/2404.12803/) (2024)
- [DreamMatcher: Appearance Matching Self-Attention for Semantically-Consistent Text-to-Image Personalization](https://mlsanity.com/paper/2402.09812/) (2024)
- [ImageBrush: Learning Visual In-Context Instructions for Exemplar-Based Image Manipulation](https://mlsanity.com/paper/2308.00906/) (2023)

## Built on by (most cited first, 30 of 63)

- [OmniEraser: Remove Objects and Their Effects in Images with Paired Video-Frame Data](https://mlsanity.com/paper/2501.07397/) (2025)
- [LBM: Latent Bridge Matching for Fast Image-to-Image Translation](https://mlsanity.com/paper/2503.07535/) (2025)
- [PlanGen: Towards Unified Layout Planning and Image Generation in Auto-Regressive Vision Language Models](https://mlsanity.com/paper/2503.10127/) (2025)
- [Diffusion Probe: Generated Image Result Prediction Using CNN Probes](https://mlsanity.com/paper/2602.23783/) (2026)
- [Precise Object and Effect Removal with Adaptive Target-Aware Attention](https://mlsanity.com/paper/2505.22636/) (2025)
- [Training-free Geometric Image Editing on Diffusion Models](https://mlsanity.com/paper/2507.23300/) (2025)
- [CustomVideoX: 3D Reference Attention Driven Dynamic Adaptation for Zero-Shot Customized Video Diffusion Transformers](https://mlsanity.com/paper/2502.06527/) (2025)
- [TC-Pad'e: Trajectory-Consistent Pad'e Approximation for Diffusion Acceleration](https://mlsanity.com/paper/2603.02943/) (2026)
- [GeoRemover: Removing Objects and Their Causal Visual Artifacts](https://mlsanity.com/paper/2509.18538/) (2025)
- [Object-WIPER : Training-Free Object and Associated Effect Removal in Videos](https://mlsanity.com/paper/2601.06391/) (2026)
- [Regression and Forecasting of U.S. Stock Returns Based on LSTM](https://mlsanity.com/paper/2502.05210/) (2025)
- [simpleposter: A simple baseline for product poster generation](https://mlsanity.com/paper/2605.08784/) (2026)
- [RePainter: Empowering E-commerce Object Removal via Spatial-matting Reinforcement Learning](https://mlsanity.com/paper/2510.07721/) (2025)
- [SpriteHand: Real-Time Versatile Hand-Object Interaction with Autoregressive Video Generation](https://mlsanity.com/paper/2512.01960/) (2025)
- [MetaVideoAgent: Automated Video-Agent Evolution for Long-Form Video Understanding](https://mlsanity.com/paper/2608.04587/) (2026)
- [ForAug: Mitigating Biases in Image Classification via Controlled Image Compositions](https://mlsanity.com/paper/2503.09399/) (2025)
- [Seeing Through Clutter: Structured 3D Scene Reconstruction via Iterative Object Removal](https://mlsanity.com/paper/2602.04053/) (2026)
- [TDRI: Two-Phase Dialogue Refinement and Co-Adaptation for Interactive Image Generation](https://mlsanity.com/paper/2503.17669/) (2025)
- [HarmonPaint: Harmonized Training-Free Diffusion Inpainting](https://mlsanity.com/paper/2507.16732/) (2025)
- [CrimEdit: Controllable Editing for Counterfactual Object Removal, Insertion, and Movement](https://mlsanity.com/paper/2509.23708/) (2025)
- [Beyond Fixed Anchors: Precisely Erasing Concepts with Sibling Exclusive Counterparts](https://mlsanity.com/paper/2510.16342/) (2025)
- [Refaccade: Editing Object with Given Reference Texture](https://mlsanity.com/paper/2512.04534/) (2025)
- [Geometric Image Editing via Effects-Sensitive In-Context Inpainting with Diffusion Transformers](https://mlsanity.com/paper/2602.08388/) (2026)
- [Language-Free Generative Editing from One Visual Example](https://mlsanity.com/paper/2603.25441/) (2026)
- [PANDORA: Pixel-wise Attention Dissolution and Latent Guidance for Zero-Shot Object Removal](https://mlsanity.com/paper/2603.27555/) (2026)
- [FlashClear: Ultra-Fast Image Content Removal via Efficient Step Distillation and Feature Caching](https://mlsanity.com/paper/2605.09003/) (2026)
- [DualEraser: Joint Video Object and Effect Removal via Balanced Text-Mask Guidance and Decoupled Locator-Preserver](https://mlsanity.com/paper/2605.30045/) (2026)
- [DORS: Dynamic Attention Routing for Diffusion-based Object Removal in Dense Scenes](https://mlsanity.com/paper/2607.16656/) (2026)
- [Hear the Scene: Audio-Enhanced Text Spotting](https://mlsanity.com/paper/2412.19504/) (2024)
- [First-place Solution for Streetscape Shop Sign Recognition Competition](https://mlsanity.com/paper/2501.02811/) (2025)

## Abstract (arXiv)

Recently, diffusion models have emerged as promising newcomers in the field of generative models, shining brightly in image generation. However, when employed for object removal tasks, they still encounter issues such as generating random artifacts and the incapacity to repaint foreground object areas with appropriate content after removal. To tackle these problems, we propose Attentive Eraser, a tuning-free method to empower pre-trained diffusion models for stable and effective object removal. Firstly, in light of the observation that the self-attention maps influence the structure and shape details of the generated images, we propose Attention Activation and Suppression (ASS), which re-engineers the self-attention mechanism within the pre-trained diffusion models based on the given mask, thereby prioritizing the background over the foreground object during the reverse generation process. Moreover, we introduce Self-Attention Redirection Guidance (SARG), which utilizes the self-attention redirected by ASS to guide the generation process, effectively removing foreground objects within the mask while simultaneously generating content that is both plausible and coherent. Experiments demonstrate the stability and effectiveness of Attentive Eraser in object removal across a variety of pre-trained diffusion models, outperforming even training-based methods. Furthermore, Attentive Eraser can be implemented in various diffusion model architectures and checkpoints, enabling excellent scalability. Code is available at https://github.com/Anonym0u3/AttentiveEraser.