# Matte Anything: Interactive Natural Image Matting with Segment Anything Models

arXiv: https://arxiv.org/abs/2306.04121 · HTML: https://mlsanity.com/paper/2306.04121/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 54 references that are in the map.
- First posted on arXiv 7 Jun 2023.

## Builds on (most cited first, 30 of 43)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [Open-Vocabulary Object Detection Using Captions](https://mlsanity.com/paper/2011.10678/) (2020)
- [OneFormer: One Transformer to Rule Universal Image Segmentation](https://mlsanity.com/paper/2211.06220/) (2022)
- [Track Anything: Segment Anything Meets Videos](https://mlsanity.com/paper/2304.11968/) (2023)
- [Inpaint Anything: Segment Anything Meets Image Inpainting](https://mlsanity.com/paper/2304.06790/) (2023)
- [Segment and Track Anything](https://mlsanity.com/paper/2305.06558/) (2023)
- [DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detection](https://mlsanity.com/paper/2209.09407/) (2022)
- [Deep Image Matting](https://mlsanity.com/paper/1703.03872/) (2017)
- [ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models](https://mlsanity.com/paper/2204.08790/) (2022)
- [Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation](https://mlsanity.com/paper/2303.07937/) (2023)
- [A Closer Look at the Explainability of Contrastive Language-Image Pre-training](https://mlsanity.com/paper/2304.05653/) (2023)
- [Indices Matter: Learning to Index for Deep Image Matting](https://mlsanity.com/paper/1908.00672/) (2019)
- [Bridging Composite and Real: Towards End-to-end Deep Image Matting](https://mlsanity.com/paper/2010.16188/) (2020)
- [Mask Guided Matting via Progressive Refinement Network](https://mlsanity.com/paper/2012.06722/) (2020)

## Built on by (most cited first, 30 of 39)

- [LangSplat: 3D Language Gaussian Splatting](https://mlsanity.com/paper/2312.16084/) (2023)
- [VRP-SAM: SAM with Visual Reference Prompt](https://mlsanity.com/paper/2402.17726/) (2024)
- [EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything Model](https://mlsanity.com/paper/2406.20076/) (2024)
- [MULAN: A Multi Layer Annotated Dataset for Controllable Text-to-Image Generation](https://mlsanity.com/paper/2404.02790/) (2024)
- [Human Hair Reconstruction with Strand-Aligned 3D Gaussians](https://mlsanity.com/paper/2409.14778/) (2024)
- [PTQ4SAM: Post-Training Quantization for Segment Anything](https://mlsanity.com/paper/2405.03144/) (2024)
- [Matting Anything](https://mlsanity.com/paper/2306.05399/) (2023)
- [Hi-SAM: Marrying Segment Anything Model for Hierarchical Text Segmentation](https://mlsanity.com/paper/2401.17904/) (2024)
- [MatAnyone: Stable Video Matting with Consistent Memory Propagation](https://mlsanity.com/paper/2501.14677/) (2025)
- [Alfie: Democratising RGBA Image Generation With No ](https://mlsanity.com/paper/2408.14826/) (2024)
- [PA-SAM: Prompt Adapter SAM for High-Quality Image Segmentation](https://mlsanity.com/paper/2401.13051/) (2024)
- [ZIM: Zero-Shot Image Matting for Anything](https://mlsanity.com/paper/2411.00626/) (2024)
- [Generating Compositional Scenes via Text-to-image RGBA Instance Generation](https://mlsanity.com/paper/2411.10913/) (2024)
- [ASAM: Boosting Segment Anything Model with Adversarial Tuning](https://mlsanity.com/paper/2405.00256/) (2024)
- [TransPixeler: Advancing Text-to-Video Generation with Transparency](https://mlsanity.com/paper/2501.03006/) (2025)
- [Diffusion for Natural Image Matting](https://mlsanity.com/paper/2312.05915/) (2023)
- [There is no SAMantics! Exploring SAM as a Backbone for Visual Understanding Tasks](https://mlsanity.com/paper/2411.15288/) (2024)
- [MatAnyone 2: Scaling Video Matting via a Learned Quality Evaluator](https://mlsanity.com/paper/2512.11782/) (2025)
- [R2-Bench: Benchmarking the Robustness of Referring Perception Models under Perturbations](https://mlsanity.com/paper/2403.04924/) (2024)
- [Industrial Segment Anything -- a Case Study in Aircraft Manufacturing, Intralogistics, Maintenance, Repair, and Overhaul](https://mlsanity.com/paper/2307.12674/) (2023)
- [SDMatte: Grafting Diffusion Models for Interactive Matting](https://mlsanity.com/paper/2508.00443/) (2025)
- [In-Context Matting](https://mlsanity.com/paper/2403.15789/) (2024)
- [Learning Trimaps via Clicks for Image Matting](https://mlsanity.com/paper/2404.00335/) (2024)
- [Towards Natural Image Matting in the Wild via Real-Scenario Prior](https://mlsanity.com/paper/2410.06593/) (2024)
- [Referring Layer Decomposition](https://mlsanity.com/paper/2602.19358/) (2026)
- [DFIMat: Decoupled Flexible Interactive Matting in Multi-Person Scenarios](https://mlsanity.com/paper/2410.09788/) (2024)
- [Object-Aware Video Matting with Cross-Frame Guidance](https://mlsanity.com/paper/2503.01262/) (2025)
- [TriSAM: Tri-Plane SAM for zero-shot cortical blood vessel segmentation in VEM images](https://mlsanity.com/paper/2401.13961/) (2024)
- [Subsurface Scattering for 3D Gaussian Splatting](https://mlsanity.com/paper/2408.12282/) (2024)
- [Structurally Disentangled Feature Fields Distillation for 3D Understanding and Editing](https://mlsanity.com/paper/2502.14789/) (2025)

## Abstract (arXiv)

Natural image matting algorithms aim to predict the transparency map (alpha-matte) with the trimap guidance. However, the production of trimap often requires significant labor, which limits the widespread application of matting algorithms on a large scale. To address the issue, we propose Matte Anything (MatAny), an interactive natural image matting model that could produce high-quality alpha-matte with various simple hints. The key insight of MatAny is to generate pseudo trimap automatically with contour and transparency prediction. In our work, we leverage vision foundation models to enhance the performance of natural image matting. Specifically, we use the segment anything model to predict high-quality contour with user interaction and an open-vocabulary detector to predict the transparency of any object. Subsequently, a pre-trained image matting model generates alpha mattes with pseudo trimaps. MatAny is the interactive matting algorithm with the most supported interaction methods and the best performance to date. It consists of orthogonal vision models without any additional training. We evaluate the performance of MatAny against several current image matting algorithms. MatAny has 58.3% improvement on MSE and 40.6% improvement on SAD compared to the previous image matting methods with simple guidance, achieving new state-of-the-art (SOTA) performance. The source codes and pre-trained models are available at https://github.com/hustvl/Matte-Anything.