# SDMatte: Grafting Diffusion Models for Interactive Matting

arXiv: https://arxiv.org/abs/2508.00443 · HTML: https://mlsanity.com/paper/2508.00443/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 46 of its 59 references that are in the map.
- First posted on arXiv 1 Aug 2025.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 46)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [Segment Anything in High Quality](https://mlsanity.com/paper/2306.01567/) (2023)
- [SegDiff: Image Segmentation with Diffusion Probabilistic Models](https://mlsanity.com/paper/2112.00390/) (2021)
- [Deep Image Matting](https://mlsanity.com/paper/1703.03872/) (2017)
- [DDP: Diffusion Model for Dense Visual Prediction](https://mlsanity.com/paper/2303.17559/) (2023)
- [Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusion](https://mlsanity.com/paper/2308.12469/) (2023)
- [MODNet: Real-Time Trimap-Free Portrait Matting via Objective Decomposition](https://mlsanity.com/paper/2011.11961/) (2020)
- [Bridging Composite and Real: Towards End-to-end Deep Image Matting](https://mlsanity.com/paper/2010.16188/) (2020)
- [Mask Guided Matting via Progressive Refinement Network](https://mlsanity.com/paper/2012.06722/) (2020)
- [What Matters When Repurposing Diffusion Models for General Dense Perception Tasks?](https://mlsanity.com/paper/2403.06090/) (2024)
- [Pyramid Grafting Network for One-Stage High Resolution Saliency Detection](https://mlsanity.com/paper/2204.05041/) (2022)
- [Privacy-Preserving Portrait Matting](https://mlsanity.com/paper/2104.14222/) (2021)
- [Deep Automatic Natural Image Matting](https://mlsanity.com/paper/2107.07235/) (2021)
- [Diffusion Models Beat GANs on Image Classification](https://mlsanity.com/paper/2307.08702/) (2023)
- [ViTMatte: Boosting Image Matting with Pretrained Plain Vision Transformers](https://mlsanity.com/paper/2305.15272/) (2023)
- [F, B, Alpha Matting](https://mlsanity.com/paper/2003.07711/) (2020)
- [Matte Anything: Interactive Natural Image Matting with Segment Anything Models](https://mlsanity.com/paper/2306.04121/) (2023)
- [DiffusionEdge: Diffusion Probabilistic Model for Crisp Edge Detection](https://mlsanity.com/paper/2401.02032/) (2024)
- [Matting Anything](https://mlsanity.com/paper/2306.05399/) (2023)
- [BetterDepth: Plug-and-Play Diffusion Refiner for Zero-Shot Monocular Depth Estimation](https://mlsanity.com/paper/2407.17952/) (2024)

## Built on by (most cited first, 4 of 4)

- [Edit2Perceive: Image Editing Diffusion Models Are Strong Dense Perceivers](https://mlsanity.com/paper/2511.18673/) (2025)
- [VideoMaMa: Mask-Guided Video Matting via Generative Prior](https://mlsanity.com/paper/2601.14255/) (2026)
- [Towards Photorealistic and Efficient Bokeh Rendering via Diffusion Framework](https://mlsanity.com/paper/2605.07429/) (2026)
- [RenderMatte: Exact-Alpha Rendering and Group-Relative Alignment for Image Matting](https://mlsanity.com/paper/2608.08487/) (2026)

## Abstract (arXiv)

Recent interactive matting methods have shown satisfactory performance in capturing the primary regions of objects, but they fall short in extracting fine-grained details in edge regions. Diffusion models trained on billions of image-text pairs, demonstrate exceptional capability in modeling highly complex data distributions and synthesizing realistic texture details, while exhibiting robust text-driven interaction capabilities, making them an attractive solution for interactive matting. To this end, we propose SDMatte, a diffusion-driven interactive matting model, with three key contributions. First, we exploit the powerful priors of diffusion models and transform the text-driven interaction capability into visual prompt-driven interaction capability to enable interactive matting. Second, we integrate coordinate embeddings of visual prompts and opacity embeddings of target objects into U-Net, enhancing SDMatte's sensitivity to spatial position information and opacity information. Third, we propose a masked self-attention mechanism that enables the model to focus on areas specified by visual prompts, leading to better performance. Extensive experiments on multiple datasets demonstrate the superior performance of our method, validating its effectiveness in interactive matting. Our code and model are available at https://github.com/vivoCameraResearch/SDMatte.