# Benchmarking Feature Upsampling Methods for Vision Foundation Models using Interactive Segmentation

arXiv: https://arxiv.org/abs/2505.02075 · HTML: https://mlsanity.com/paper/2505.02075/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 70 references that are in the map.
- First posted on arXiv 4 May 2025.

## Builds on (most cited first, 30 of 50)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [Exploring Plain Vision Transformer Backbones for Object Detection](https://mlsanity.com/paper/2203.16527/) (2022)
- [DiffEdit: Diffusion-based semantic image editing with mask guidance](https://mlsanity.com/paper/2210.11427/) (2022)
- [Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality](https://mlsanity.com/paper/2204.03162/) (2022)
- [PointRend: Image Segmentation as Rendering](https://mlsanity.com/paper/1912.08193/) (2019)
- [Learning Deconvolution Network for Semantic Segmentation](https://mlsanity.com/paper/1505.04366/) (2015)
- [Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP](https://mlsanity.com/paper/2308.02487/) (2023)
- [A guide to convolution arithmetic for deep learning](https://mlsanity.com/paper/1603.07285/) (2016)
- [Probing the 3D Awareness of Visual Foundation Models](https://mlsanity.com/paper/2404.08636/) (2024)
- [AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into One](https://mlsanity.com/paper/2312.06709/) (2023)
- [Reviving Iterative Training with Mask Guidance for Interactive Segmentation](https://mlsanity.com/paper/2102.06583/) (2021)
- [Deep Interactive Object Selection](https://mlsanity.com/paper/1603.04042/) (2016)
- [ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback](https://mlsanity.com/paper/2404.07987/) (2024)
- [LEDITS++: Limitless Image Editing using Text-to-Image Models](https://mlsanity.com/paper/2311.16711/) (2023)
- [CARAFE: Content-Aware ReAssembly of FEatures](https://mlsanity.com/paper/1905.02188/) (2019)
- [FeatUp: A Model-Agnostic Framework for Features at Any Resolution](https://mlsanity.com/paper/2403.10516/) (2024)
- [SimpleClick: Interactive Image Segmentation with Simple Vision Transformers](https://mlsanity.com/paper/2210.11006/) (2022)

## Built on by (most cited first, 1 of 1)

- [Maybe you don't need a U-Net: convolutional feature upsampling for materials micrograph segmentation](https://mlsanity.com/paper/2508.21529/) (2025)

## Abstract (arXiv)

Vision Foundation Models (VFMs) are large-scale, pre-trained models that serve as general-purpose backbones for various computer vision tasks. As VFMs' popularity grows, there is an increasing interest in understanding their effectiveness for dense prediction tasks. However, VFMs typically produce low-resolution features, limiting their direct applicability in this context. One way to tackle this limitation is by employing a task-agnostic feature upsampling module that refines VFM features resolution. To assess the effectiveness of this approach, we investigate Interactive Segmentation (IS) as a novel benchmark for evaluating feature upsampling methods on VFMs. Due to its inherent multimodal input, consisting of an image and a set of user-defined clicks, as well as its dense mask output, IS creates a challenging environment that demands comprehensive visual scene understanding. Our benchmarking experiments show that selecting appropriate upsampling strategies significantly improves VFM features quality. The code is released at https://github.com/havrylovv/iSegProbe