# MatteFormer: Transformer-Based Image Matting via Prior-Tokens

arXiv: https://arxiv.org/abs/2203.15662 · HTML: https://mlsanity.com/paper/2203.15662/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 57 references that are in the map.
- First posted on arXiv 29 Mar 2022.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [Pre-Trained Image Processing Transformer](https://mlsanity.com/paper/2012.00364/) (2020)
- [Transformer in Transformer](https://mlsanity.com/paper/2103.00112/) (2021)
- [CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows](https://mlsanity.com/paper/2107.00652/) (2021)
- [Visual Transformers: Token-based Image Representation and Processing for Computer Vision](https://mlsanity.com/paper/2006.03677/) (2020)
- [Focal Self-attention for Local-Global Interactions in Vision Transformers](https://mlsanity.com/paper/2107.00641/) (2021)
- [LocalViT: Analyzing Locality in Vision Transformers](https://mlsanity.com/paper/2104.05707/) (2021)
- [Scaling Local Self-Attention for Parameter Efficient Visual Backbones](https://mlsanity.com/paper/2103.12731/) (2021)
- [Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding](https://mlsanity.com/paper/2103.15358/) (2021)
- [Real-Time High-Resolution Background Matting](https://mlsanity.com/paper/2012.07810/) (2020)
- [Deep Image Matting](https://mlsanity.com/paper/1703.03872/) (2017)
- [High-Fidelity Pluralistic Image Completion with Transformers](https://mlsanity.com/paper/2103.14031/) (2021)
- [TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?](https://mlsanity.com/paper/2106.11297/) (2021)
- [Background Matting: The World is Your Green Screen](https://mlsanity.com/paper/2004.00626/) (2020)
- [Indices Matter: Learning to Index for Deep Image Matting](https://mlsanity.com/paper/1908.00672/) (2019)
- [Mask Guided Matting via Progressive Refinement Network](https://mlsanity.com/paper/2012.06722/) (2020)
- [Toward Transformer-Based Object Detection](https://mlsanity.com/paper/2012.09958/) (2020)

## Built on by (most cited first, 30 of 48)

- [Transformer-Based Visual Segmentation: A Survey](https://mlsanity.com/paper/2304.09854/) (2023)
- [ViTMatte: Boosting Image Matting with Pretrained Plain Vision Transformers](https://mlsanity.com/paper/2305.15272/) (2023)
- [X-Paste: Revisiting Scalable Copy-Paste for Instance Segmentation using CLIP and StableDiffusion](https://mlsanity.com/paper/2212.03863/) (2022)
- [Neural-PBIR Reconstruction of Shape, Material, and Illumination](https://mlsanity.com/paper/2304.13445/) (2023)
- [Matte Anything: Interactive Natural Image Matting with Segment Anything Models](https://mlsanity.com/paper/2306.04121/) (2023)
- [Text2Layer: Layered Image Generation using Latent Diffusion Model](https://mlsanity.com/paper/2307.09781/) (2023)
- [Matting Anything](https://mlsanity.com/paper/2306.05399/) (2023)
- [CLE Diffusion: Controllable Light Enhancement Diffusion Model](https://mlsanity.com/paper/2308.06725/) (2023)
- [VMFormer: End-to-End Video Matting with Transformer](https://mlsanity.com/paper/2208.12801/) (2022)
- [ZIM: Zero-Shot Image Matting for Anything](https://mlsanity.com/paper/2411.00626/) (2024)
- [Deep Image Matting: A Comprehensive Survey](https://mlsanity.com/paper/2304.04672/) (2023)
- [Diffusion for Natural Image Matting](https://mlsanity.com/paper/2312.05915/) (2023)
- [Revisiting Context Aggregation for Image Matting](https://mlsanity.com/paper/2304.01171/) (2023)
- [Adversarially-Guided Portrait Matting](https://mlsanity.com/paper/2305.02981/) (2023)
- [dugMatting: Decomposed-Uncertainty-Guided Matting](https://mlsanity.com/paper/2306.01452/) (2023)
- [Boosting General Trimap-free Matting in the Real-World Image](https://mlsanity.com/paper/2405.17916/) (2024)
- [Training Matting Models without Alpha Labels](https://mlsanity.com/paper/2408.10539/) (2024)
- [Out-of-domain GAN inversion via Invertibility Decomposition for Photo-Realistic Human Face Manipulation](https://mlsanity.com/paper/2212.09262/) (2022)
- [Towards Label-Efficient Human Matting: A Simple Baseline for Weakly Semi-Supervised Trimap-Free Human Matting](https://mlsanity.com/paper/2404.00921/) (2024)
- [PViT: Prior-augmented Vision Transformer for Out-of-distribution Detection](https://mlsanity.com/paper/2410.20631/) (2024)
- [Referring Layer Decomposition](https://mlsanity.com/paper/2602.19358/) (2026)
- [CineMatte: Background Matting for Virtual Production and Beyond](https://mlsanity.com/paper/2605.18328/) (2026)
- [Exploring the Interactive Guidance for Unified and Effective Image Matting](https://mlsanity.com/paper/2205.08324/) (2022)
- [Privileged Prior Information Distillation for Image Matting](https://mlsanity.com/paper/2211.14036/) (2022)
- [Infusing Definiteness into Randomness: Rethinking Composition Styles for Deep Image Matting](https://mlsanity.com/paper/2212.13517/) (2022)
- [Disentangled Pre-training for Image Matting](https://mlsanity.com/paper/2304.00784/) (2023)
- [DiffusionMat: Alpha Matting as Sequential Refinement Learning](https://mlsanity.com/paper/2311.13535/) (2023)
- [TETRIS: Towards Exploring the Robustness of Interactive Segmentation](https://mlsanity.com/paper/2402.06132/) (2024)
- [Zippo: Zipping Color and Transparency Distributions into a Single Diffusion Model](https://mlsanity.com/paper/2403.11077/) (2024)
- [Deep Image Composition Meets Image Forgery](https://mlsanity.com/paper/2404.02897/) (2024)

## Abstract (arXiv)

In this paper, we propose a transformer-based image matting model called MatteFormer, which takes full advantage of trimap information in the transformer block. Our method first introduces a prior-token which is a global representation of each trimap region (e.g. foreground, background and unknown). These prior-tokens are used as global priors and participate in the self-attention mechanism of each block. Each stage of the encoder is composed of PAST (Prior-Attentive Swin Transformer) block, which is based on the Swin Transformer block, but differs in a couple of aspects: 1) It has PA-WSA (Prior-Attentive Window Self-Attention) layer, performing self-attention not only with spatial-tokens but also with prior-tokens. 2) It has prior-memory which saves prior-tokens accumulatively from the previous blocks and transfers them to the next block. We evaluate our MatteFormer on the commonly used image matting datasets: Composition-1k and Distinctions-646. Experiment results show that our proposed method achieves state-of-the-art performance with a large margin. Our codes are available at https://github.com/webtoon/matteformer.