# Learning Affinity-Aware Upsampling for Deep Image Matting

arXiv: https://arxiv.org/abs/2011.14288 · HTML: https://mlsanity.com/paper/2011.14288/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Nov 2020.

## Built on by (most cited first, 30 of 37)

- [FeatUp: A Model-Agnostic Framework for Features at Any Resolution](https://mlsanity.com/paper/2403.10516/) (2024)
- [Learning to Upsample by Learning to Sample](https://mlsanity.com/paper/2308.15085/) (2023)
- [ViTMatte: Boosting Image Matting with Pretrained Plain Vision Transformers](https://mlsanity.com/paper/2305.15272/) (2023)
- [Frequency-aware Feature Fusion for Dense Image Prediction](https://mlsanity.com/paper/2408.12879/) (2024)
- [SAPA: Similarity-Aware Point Affiliation for Feature Upsampling](https://mlsanity.com/paper/2209.12866/) (2022)
- [LiFT: A Surprisingly Simple Lightweight Feature Transform for Dense ViT Descriptors](https://mlsanity.com/paper/2403.14625/) (2024)
- [Human Instance Matting via Mutual Guidance and Multi-Instance Refinement](https://mlsanity.com/paper/2205.10767/) (2022)
- [FADE: Fusing the Assets of Decoder and Encoder for Task-Agnostic Upsampling](https://mlsanity.com/paper/2207.10392/) (2022)
- [LoftUp: Learning a Coordinate-Based Feature Upsampler for Vision Foundation Models](https://mlsanity.com/paper/2504.14032/) (2025)
- [Boosting Robustness of Image Matting with Context Assembling and Strong Data Augmentation](https://mlsanity.com/paper/2201.06889/) (2022)
- [Deep Image Matting: A Comprehensive Survey](https://mlsanity.com/paper/2304.04672/) (2023)
- [Diffusion for Natural Image Matting](https://mlsanity.com/paper/2312.05915/) (2023)
- [EISeg: An Efficient Interactive Segmentation Tool based on PaddlePaddle](https://mlsanity.com/paper/2210.08788/) (2022)
- [A Refreshed Similarity-based Upsampler for Direct High-Ratio Feature Upsampling](https://mlsanity.com/paper/2407.02283/) (2024)
- [Revisiting Context Aggregation for Image Matting](https://mlsanity.com/paper/2304.01171/) (2023)
- [Hierarchical and Progressive Image Matting](https://mlsanity.com/paper/2210.06906/) (2022)
- [Butter: Frequency Consistency and Hierarchical Fusion for Autonomous Driving Object Detection](https://mlsanity.com/paper/2507.13373/) (2025)
- [Alpha Matte Generation from Single Input for Portrait Matting](https://mlsanity.com/paper/2106.03210/) (2021)
- [Guided Linear Upsampling](https://mlsanity.com/paper/2307.09582/) (2023)
- [FADE: A Task-Agnostic Upsampling Operator for Encoder-Decoder Architectures](https://mlsanity.com/paper/2407.13500/) (2024)
- [CineMatte: Background Matting for Virtual Production and Beyond](https://mlsanity.com/paper/2605.18328/) (2026)
- [M2MRF: Many-to-Many Reassembly of Features for Tiny Lesion Segmentation in Fundus Images](https://mlsanity.com/paper/2111.00193/) (2021)
- [Trimap-guided Feature Mining and Fusion Network for Natural Image Matting](https://mlsanity.com/paper/2112.00510/) (2021)
- [Improving Deep Image Matting via Local Smoothness Assumption](https://mlsanity.com/paper/2112.13809/) (2021)
- [Infusing Definiteness into Randomness: Rethinking Composition Styles for Deep Image Matting](https://mlsanity.com/paper/2212.13517/) (2022)
- [Event USKT : U-State Space Model in Knowledge Transfer for Event Cameras](https://mlsanity.com/paper/2411.15276/) (2024)
- [Semantic-guided Automatic Natural Image Matting with Trimap Generation Network and Light-weight Non-local Attention](https://mlsanity.com/paper/2103.17020/) (2021)
- [TransMatting: Tri-token Equipped Transformer Model for Image Matting](https://mlsanity.com/paper/2303.06476/) (2023)
- [On Point Affiliation in Feature Upsampling](https://mlsanity.com/paper/2307.08198/) (2023)
- [Benchmarking Feature Upsampling Methods for Vision Foundation Models using Interactive Segmentation](https://mlsanity.com/paper/2505.02075/) (2025)

## Abstract (arXiv)

We show that learning affinity in upsampling provides an effective and efficient approach to exploit pairwise interactions in deep networks. Second-order features are commonly used in dense prediction to build adjacent relations with a learnable module after upsampling such as non-local blocks. Since upsampling is essential, learning affinity in upsampling can avoid additional propagation layers, offering the potential for building compact models. By looking at existing upsampling operators from a unified mathematical perspective, we generalize them into a second-order form and introduce Affinity-Aware Upsampling (A2U) where upsampling kernels are generated using a light-weight lowrank bilinear model and are conditioned on second-order features. Our upsampling operator can also be extended to downsampling. We discuss alternative implementations of A2U and verify their effectiveness on two detail-sensitive tasks: image reconstruction on a toy dataset; and a largescale image matting task where affinity-based ideas constitute mainstream matting approaches. In particular, results on the Composition-1k matting dataset show that A2U achieves a 14% relative improvement in the SAD metric against a strong baseline with negligible increase of parameters (<0.5%). Compared with the state-of-the-art matting network, we achieve 8% higher performance with only 40% model complexity.