# SAPA: Similarity-Aware Point Affiliation for Feature Upsampling

arXiv: https://arxiv.org/abs/2209.12866 · HTML: https://mlsanity.com/paper/2209.12866/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 44 references that are in the map.
- First posted on arXiv 26 Sep 2022.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation](https://mlsanity.com/paper/1907.10326/) (2019)
- [Deep Image Matting](https://mlsanity.com/paper/1703.03872/) (2017)
- [CARAFE: Content-Aware ReAssembly of FEatures](https://mlsanity.com/paper/1905.02188/) (2019)
- [FastDepth: Fast Monocular Depth Estimation on Embedded Systems](https://mlsanity.com/paper/1903.03273/) (2019)
- [Indices Matter: Learning to Index for Deep Image Matting](https://mlsanity.com/paper/1908.00672/) (2019)
- [Learning Affinity-Aware Upsampling for Deep Image Matting](https://mlsanity.com/paper/2011.14288/) (2020)
- [Deep attention-based classification network for robust depth prediction](https://mlsanity.com/paper/1807.03959/) (2018)
- [Decoders Matter for Semantic Segmentation: Data-Dependent Decoding Enables Flexible Feature Aggregation](https://mlsanity.com/paper/1903.02120/) (2019)
- [FADE: Fusing the Assets of Decoder and Encoder for Task-Agnostic Upsampling](https://mlsanity.com/paper/2207.10392/) (2022)
- [The Devil is in the Decoder: Classification, Regression and GANs](https://mlsanity.com/paper/1707.05847/) (2017)
- [CARAFE++: Unified Content-Aware ReAssembly of FEatures](https://mlsanity.com/paper/2012.04733/) (2020)

## Built on by (most cited first, 30 of 32)

- [FeatUp: A Model-Agnostic Framework for Features at Any Resolution](https://mlsanity.com/paper/2403.10516/) (2024)
- [Learning to Upsample by Learning to Sample](https://mlsanity.com/paper/2308.15085/) (2023)
- [Frequency-aware Feature Fusion for Dense Image Prediction](https://mlsanity.com/paper/2408.12879/) (2024)
- [AnyUp: Universal Feature Upsampling](https://mlsanity.com/paper/2510.12764/) (2025)
- [LiFT: A Surprisingly Simple Lightweight Feature Transform for Dense ViT Descriptors](https://mlsanity.com/paper/2403.14625/) (2024)
- [SegEarth-OV: Towards Training-Free Open-Vocabulary Segmentation for Remote Sensing Images](https://mlsanity.com/paper/2410.01768/) (2024)
- [LoftUp: Learning a Coordinate-Based Feature Upsampler for Vision Foundation Models](https://mlsanity.com/paper/2504.14032/) (2025)
- [JAFAR: Jack up Any Feature at Any Resolution](https://mlsanity.com/paper/2506.11136/) (2025)
- [Annotation-Free Open-Vocabulary Segmentation for Remote-Sensing Images](https://mlsanity.com/paper/2508.18067/) (2025)
- [A Refreshed Similarity-based Upsampler for Direct High-Ratio Feature Upsampling](https://mlsanity.com/paper/2407.02283/) (2024)
- [FeatSharp: Your Vision Model Features, Sharper](https://mlsanity.com/paper/2502.16025/) (2025)
- [NAF: Zero-Shot Feature Upsampling via Neighborhood Attention Filtering](https://mlsanity.com/paper/2511.18452/) (2025)
- [DenseSR: Image Shadow Removal as Dense Prediction](https://mlsanity.com/paper/2507.16472/) (2025)
- [FADE: A Task-Agnostic Upsampling Operator for Encoder-Decoder Architectures](https://mlsanity.com/paper/2407.13500/) (2024)
- [FreqPDE: Rethinking Positional Depth Embedding for Multi-View 3D Object Detection Transformers](https://mlsanity.com/paper/2510.15385/) (2025)
- [UPLiFT: Efficient Pixel-Dense Feature Upsampling with Local Attenders](https://mlsanity.com/paper/2601.17950/) (2026)
- [On Point Affiliation in Feature Upsampling](https://mlsanity.com/paper/2307.08198/) (2023)
- [Benchmarking Feature Upsampling Methods for Vision Foundation Models using Interactive Segmentation](https://mlsanity.com/paper/2505.02075/) (2025)
- [Dynamic Snake Upsampling Operater and Boundary-Skeleton Weighted Loss for Tubular Structure Segmentation](https://mlsanity.com/paper/2505.08525/) (2025)
- [First Place Solution to the MLCAS 2025 GWFSS Challenge: The Devil is in the Detail and Minority](https://mlsanity.com/paper/2508.17305/) (2025)
- [Lighten CARAFE: Dynamic Lightweight Upsampling with Guided Reassemble Kernels](https://mlsanity.com/paper/2410.22139/) (2024)
- [LDA-AQU: Adaptive Query-guided Upsampling via Local Deformable Attention](https://mlsanity.com/paper/2411.19585/) (2024)
- [Fourier-Guided Attention Upsampling for Image Super-Resolution](https://mlsanity.com/paper/2508.10616/) (2025)
- [DyGLNet: Hybrid Global-Local Feature Fusion with Dynamic Upsampling for Medical Image Segmentation](https://mlsanity.com/paper/2509.12763/) (2025)
- [Tracking spatial temporal details in ultrasound long video via wavelet analysis and memory bank](https://mlsanity.com/paper/2512.15066/) (2025)
- [DTC: A Deformable Transposed Convolution Module for Medical Image Segmentation](https://mlsanity.com/paper/2601.17939/) (2026)
- [Excite, Attend and Segment (EASe): Domain-Agnostic Fine-Grained Mask Discovery with Feature Calibration and Self-Supervised Upsampling](https://mlsanity.com/paper/2604.00276/) (2026)
- [A3-FPN: Asymptotic Content-Aware Pyramid Attention Network for Dense Visual Prediction](https://mlsanity.com/paper/2604.10210/) (2026)
- [Weighted Reverse Convolution for Feature Upsampling](https://mlsanity.com/paper/2605.17472/) (2026)
- [ViT-Up: Faithful Feature Upsampling for Vision Transformers](https://mlsanity.com/paper/2606.14024/) (2026)

## Abstract (arXiv)

We introduce point affiliation into feature upsampling, a notion that describes the affiliation of each upsampled point to a semantic cluster formed by local decoder feature points with semantic similarity. By rethinking point affiliation, we present a generic formulation for generating upsampling kernels. The kernels encourage not only semantic smoothness but also boundary sharpness in the upsampled feature maps. Such properties are particularly useful for some dense prediction tasks such as semantic segmentation. The key idea of our formulation is to generate similarity-aware kernels by comparing the similarity between each encoder feature point and the spatially associated local region of decoder features. In this way, the encoder feature point can function as a cue to inform the semantic cluster of upsampled feature points. To embody the formulation, we further instantiate a lightweight upsampling operator, termed Similarity-Aware Point Affiliation (SAPA), and investigate its variants. SAPA invites consistent performance improvements on a number of dense prediction tasks, including semantic segmentation, object detection, depth estimation, and image matting. Code is available at: https://github.com/poppinace/sapa