# ViTMatte: Boosting Image Matting with Pretrained Plain Vision Transformers

arXiv: https://arxiv.org/abs/2305.15272 · HTML: https://mlsanity.com/paper/2305.15272/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 69 references that are in the map.
- First posted on arXiv 24 May 2023.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [Exploring Plain Vision Transformer Backbones for Object Detection](https://mlsanity.com/paper/2203.16527/) (2022)
- [EVA: Exploring the Limits of Masked Visual Representation Learning at Scale](https://mlsanity.com/paper/2211.07636/) (2022)
- [CoAtNet: Marrying Convolution and Attention for All Data Sizes](https://mlsanity.com/paper/2106.04803/) (2021)
- [ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation](https://mlsanity.com/paper/2204.12484/) (2022)
- [Early Convolutions Help Transformers See Better](https://mlsanity.com/paper/2106.14881/) (2021)
- [EVA-02: A Visual Representation for Neon Genesis](https://mlsanity.com/paper/2303.11331/) (2023)
- [Rethinking Spatial Dimensions of Vision Transformers](https://mlsanity.com/paper/2103.16302/) (2021)
- [Deep Image Matting](https://mlsanity.com/paper/1703.03872/) (2017)
- [Advancing Plain Vision Transformer Towards Remote Sensing Foundation Model](https://mlsanity.com/paper/2208.03987/) (2022)
- [Robust High-Resolution Video Matting with Temporal Guidance](https://mlsanity.com/paper/2108.11515/) (2021)
- [Indices Matter: Learning to Index for Deep Image Matting](https://mlsanity.com/paper/1908.00672/) (2019)
- [Mask Guided Matting via Progressive Refinement Network](https://mlsanity.com/paper/2012.06722/) (2020)

## Built on by (most cited first, 30 of 57)

- [Optimal Transport Aggregation for Visual Place Recognition](https://mlsanity.com/paper/2311.15937/) (2023)
- [Transparent Image Layer Diffusion using Latent Transparency](https://mlsanity.com/paper/2402.17113/) (2024)
- [PosterMaker: Towards High-Quality Product Poster Generation with Accurate Text Rendering](https://mlsanity.com/paper/2504.06632/) (2025)
- [EndoDAC: Efficient Adapting Foundation Model for Self-Supervised Depth Estimation from Any Endoscopic Camera](https://mlsanity.com/paper/2405.08672/) (2024)
- [Matte Anything: Interactive Natural Image Matting with Segment Anything Models](https://mlsanity.com/paper/2306.04121/) (2023)
- [MULAN: A Multi Layer Annotated Dataset for Controllable Text-to-Image Generation](https://mlsanity.com/paper/2404.02790/) (2024)
- [MatAnyone: Stable Video Matting with Consistent Memory Propagation](https://mlsanity.com/paper/2501.14677/) (2025)
- [TransAnimate: Taming Layer Diffusion to Generate RGBA Video](https://mlsanity.com/paper/2503.17934/) (2025)
- [LayerD: Decomposing Raster Graphic Designs into Layers](https://mlsanity.com/paper/2509.25134/) (2025)
- [Alfie: Democratising RGBA Image Generation With No ](https://mlsanity.com/paper/2408.14826/) (2024)
- [Erase Diffusion: Empowering Object Removal Through Calibrating Diffusion Pathways](https://mlsanity.com/paper/2503.07026/) (2025)
- [ZIM: Zero-Shot Image Matting for Anything](https://mlsanity.com/paper/2411.00626/) (2024)
- [ViTGaze: Gaze Following with Interaction Features in Vision Transformers](https://mlsanity.com/paper/2403.12778/) (2024)
- [Exploring the Synergies of Hybrid CNNs and ViTs Architectures for Computer Vision: A survey](https://mlsanity.com/paper/2402.02941/) (2024)
- [Diffusion for Natural Image Matting](https://mlsanity.com/paper/2312.05915/) (2023)
- [TransPixeler: Advancing Text-to-Video Generation with Transparency](https://mlsanity.com/paper/2501.03006/) (2025)
- [GroundingSuite: Measuring Complex Multi-Granular Pixel Grounding](https://mlsanity.com/paper/2503.10596/) (2025)
- [BEN: Using Confidence-Guided Matting for Dichotomous Image Segmentation](https://mlsanity.com/paper/2501.06230/) (2025)
- [LayerEdit: Disentangled Multi-Object Editing via Conflict-Aware Multi-Layer Learning](https://mlsanity.com/paper/2511.08251/) (2025)
- [MatAnyone 2: Scaling Video Matting via a Learned Quality Evaluator](https://mlsanity.com/paper/2512.11782/) (2025)
- [Repositioning the Subject within Image](https://mlsanity.com/paper/2401.16861/) (2024)
- [Improving Object Detector Training on Synthetic Data by Starting With a Strong Baseline Methodology](https://mlsanity.com/paper/2405.19822/) (2024)
- [ChangeViT: Unleashing Plain Vision Transformers for Change Detection](https://mlsanity.com/paper/2406.12847/) (2024)
- [ImmerseGen: Agent-Guided Immersive World Generation with Alpha-Textured Proxies](https://mlsanity.com/paper/2506.14315/) (2025)
- [Edit2Perceive: Image Editing Diffusion Models Are Strong Dense Perceivers](https://mlsanity.com/paper/2511.18673/) (2025)
- [Training Matting Models without Alpha Labels](https://mlsanity.com/paper/2408.10539/) (2024)
- [Data-Efficient Inference of Neural Fluid Fields via SciML Foundation Model](https://mlsanity.com/paper/2412.13897/) (2024)
- [SDMatte: Grafting Diffusion Models for Interactive Matting](https://mlsanity.com/paper/2508.00443/) (2025)
- [In-Context Matting](https://mlsanity.com/paper/2403.15789/) (2024)
- [Towards Natural Image Matting in the Wild via Real-Scenario Prior](https://mlsanity.com/paper/2410.06593/) (2024)

## Abstract (arXiv)

Recently, plain vision Transformers (ViTs) have shown impressive performance on various computer vision tasks, thanks to their strong modeling capacity and large-scale pretraining. However, they have not yet conquered the problem of image matting. We hypothesize that image matting could also be boosted by ViTs and present a new efficient and robust ViT-based matting system, named ViTMatte. Our method utilizes (i) a hybrid attention mechanism combined with a convolution neck to help ViTs achieve an excellent performance-computation trade-off in matting tasks. (ii) Additionally, we introduce the detail capture module, which just consists of simple lightweight convolutions to complement the detailed information required by matting. To the best of our knowledge, ViTMatte is the first work to unleash the potential of ViT on image matting with concise adaptation. It inherits many superior properties from ViT to matting, including various pretraining strategies, concise architecture design, and flexible inference strategies. We evaluate ViTMatte on Composition-1k and Distinctions-646, the most commonly used benchmark for image matting, our method achieves state-of-the-art performance and outperforms prior matting works by a large margin.