# Matting by Generation

arXiv: https://arxiv.org/abs/2407.21017 · HTML: https://mlsanity.com/paper/2407.21017/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 53 references that are in the map.
- First posted on arXiv 30 Jul 2024.

## Builds on (most cited first, 29 of 29)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation](https://mlsanity.com/paper/2302.08113/) (2023)
- [ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shifting](https://mlsanity.com/paper/2307.12348/) (2023)
- [DiffIR: Efficient Diffusion Model for Image Restoration](https://mlsanity.com/paper/2303.09472/) (2023)
- [SnapFusion: Text-to-Image Diffusion Model on Mobile Devices within Two Seconds](https://mlsanity.com/paper/2306.00980/) (2023)
- [Generative Diffusion Prior for Unified Image Restoration and Enhancement](https://mlsanity.com/paper/2304.01247/) (2023)
- [MODNet: Real-Time Trimap-Free Portrait Matting via Objective Decomposition](https://mlsanity.com/paper/2011.11961/) (2020)
- [Background Matting: The World is Your Green Screen](https://mlsanity.com/paper/2004.00626/) (2020)
- [Indices Matter: Learning to Index for Deep Image Matting](https://mlsanity.com/paper/1908.00672/) (2019)
- [Bridging Composite and Real: Towards End-to-end Deep Image Matting](https://mlsanity.com/paper/2010.16188/) (2020)
- [Mask Guided Matting via Progressive Refinement Network](https://mlsanity.com/paper/2012.06722/) (2020)
- [Context-Aware Image Matting for Simultaneous Foreground and Alpha Estimation](https://mlsanity.com/paper/1909.09725/) (2019)
- [Semantic Human Matting](https://mlsanity.com/paper/1809.01354/) (2018)
- [Privacy-Preserving Portrait Matting](https://mlsanity.com/paper/2104.14222/) (2021)
- [Information-Flow Matting](https://mlsanity.com/paper/1707.05055/) (2017)
- [Semantic Image Matting](https://mlsanity.com/paper/2104.08201/) (2021)
- [Boosting Semantic Human Matting with Coarse Annotations](https://mlsanity.com/paper/2004.04955/) (2020)
- [Peekaboo: Text to Image Diffusion Models are Zero-Shot Segmentors](https://mlsanity.com/paper/2211.13224/) (2022)
- [Rethinking Portrait Matting with Privacy Preserving](https://mlsanity.com/paper/2203.16828/) (2022)
- [GANimator: Neural Motion Synthesis from a Single Sequence](https://mlsanity.com/paper/2205.02625/) (2022)
- [Improved Image Matting via Real-time User Clicks and Uncertainty Estimation](https://mlsanity.com/paper/2012.08323/) (2020)
- [Fast Multi-Level Foreground Estimation](https://mlsanity.com/paper/2006.14970/) (2020)
- [dugMatting: Decomposed-Uncertainty-Guided Matting](https://mlsanity.com/paper/2306.01452/) (2023)
- [Magenta Green Screen: Spectrally Multiplexed Alpha Matting with Deep Colorization](https://mlsanity.com/paper/2306.13702/) (2023)

## Built on by (most cited first, 13 of 13)

- [Generative Omnimatte: Learning to Decompose Video into Layers](https://mlsanity.com/paper/2411.16683/) (2024)
- [TransPixeler: Advancing Text-to-Video Generation with Transparency](https://mlsanity.com/paper/2501.03006/) (2025)
- [LightsOut: Diffusion-based Outpainting for Enhanced Lens Flare Removal](https://mlsanity.com/paper/2510.15868/) (2025)
- [SDMatte: Grafting Diffusion Models for Interactive Matting](https://mlsanity.com/paper/2508.00443/) (2025)
- [Generative Refocusing: Flexible Defocus Control from a Single Image](https://mlsanity.com/paper/2512.16923/) (2025)
- [Generative Video Matting](https://mlsanity.com/paper/2508.07905/) (2025)
- [BiVM: Accurate Binarized Neural Network for Efficient Video Matting](https://mlsanity.com/paper/2507.04456/) (2025)
- [LawDIS: Language-Window-based Controllable Dichotomous Image Segmentation](https://mlsanity.com/paper/2508.01152/) (2025)
- [EasyOmnimatte: Taming Pretrained Inpainting Diffusion Models for End-to-End Video Layered Decomposition](https://mlsanity.com/paper/2512.21865/) (2025)
- [Capture Stage Matting: Challenges, Approaches, and Solutions for Offline and Real-Time Processing](https://mlsanity.com/paper/2507.07623/) (2025)
- [HDR Reconstruction Boosting with Training-Free and Exposure-Consistent Diffusion](https://mlsanity.com/paper/2602.19706/) (2026)
- [Reflection Separation from a Single Image via Joint Latent Diffusion](https://mlsanity.com/paper/2606.04107/) (2026)
- [Parallax Portrait Matting](https://mlsanity.com/paper/2607.11205/) (2026)

## Abstract (arXiv)

This paper introduces an innovative approach for image matting that redefines the traditional regression-based task as a generative modeling challenge. Our method harnesses the capabilities of latent diffusion models, enriched with extensive pre-trained knowledge, to regularize the matting process. We present novel architectural innovations that empower our model to produce mattes with superior resolution and detail. The proposed method is versatile and can perform both guidance-free and guidance-based image matting, accommodating a variety of additional cues. Our comprehensive evaluation across three benchmark datasets demonstrates the superior performance of our approach, both quantitatively and qualitatively. The results not only reflect our method's robust effectiveness but also highlight its ability to generate visually compelling mattes that approach photorealistic quality. The project page for this paper is available at https://lightchaserx.github.io/matting-by-generation/