# LoftUp: Learning a Coordinate-Based Feature Upsampler for Vision Foundation Models

arXiv: https://arxiv.org/abs/2504.14032 · HTML: https://mlsanity.com/paper/2504.14032/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 64 references that are in the map.
- First posted on arXiv 18 Apr 2025.

## Builds on (most cited first, 30 of 47)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Exploring Plain Vision Transformer Backbones for Object Detection](https://mlsanity.com/paper/2203.16527/) (2022)
- [Learning Continuous Image Representation with Local Implicit Image Function](https://mlsanity.com/paper/2012.09161/) (2020)
- [GLaMM: Pixel Grounding Large Multimodal Model](https://mlsanity.com/paper/2311.03356/) (2023)
- [PointRend: Image Segmentation as Rendering](https://mlsanity.com/paper/1912.08193/) (2019)
- [Learning Deconvolution Network for Semantic Segmentation](https://mlsanity.com/paper/1505.04366/) (2015)
- [Splatter Image: Ultra-Fast Single-View 3D Reconstruction](https://mlsanity.com/paper/2312.13150/) (2023)
- [Unsupervised Semantic Segmentation by Distilling Feature Correspondences](https://mlsanity.com/paper/2203.08414/) (2022)
- [Space-Time Correspondence as a Contrastive Random Walk](https://mlsanity.com/paper/2006.14613/) (2020)
- [Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP](https://mlsanity.com/paper/2308.02487/) (2023)
- [A guide to convolution arithmetic for deep learning](https://mlsanity.com/paper/1603.07285/) (2016)
- [Probing the 3D Awareness of Visual Foundation Models](https://mlsanity.com/paper/2404.08636/) (2024)
- [AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into One](https://mlsanity.com/paper/2312.06709/) (2023)
- [Self-Supervised Transformers for Unsupervised Object Discovery using Normalized Cut](https://mlsanity.com/paper/2202.11539/) (2022)
- [Reviving Iterative Training with Mask Guidance for Interactive Segmentation](https://mlsanity.com/paper/2102.06583/) (2021)

## Built on by (most cited first, 23 of 23)

- [AnyUp: Universal Feature Upsampling](https://mlsanity.com/paper/2510.12764/) (2025)
- [JAFAR: Jack up Any Feature at Any Resolution](https://mlsanity.com/paper/2506.11136/) (2025)
- [CoWTracker: Tracking by Warping instead of Correlation](https://mlsanity.com/paper/2602.04877/) (2026)
- [Upsample Anything: A Simple and Hard to Beat Baseline for Feature Upsampling](https://mlsanity.com/paper/2511.16301/) (2025)
- [NAF: Zero-Shot Feature Upsampling via Neighborhood Attention Filtering](https://mlsanity.com/paper/2511.18452/) (2025)
- [Maybe you don't need a U-Net: convolutional feature upsampling for materials micrograph segmentation](https://mlsanity.com/paper/2508.21529/) (2025)
- [FlowFeat: Pixel-Dense Embedding of Motion Profiles](https://mlsanity.com/paper/2511.07696/) (2025)
- [UPLiFT: Efficient Pixel-Dense Feature Upsampling with Local Attenders](https://mlsanity.com/paper/2601.17950/) (2026)
- [Benchmarking Feature Upsampling Methods for Vision Foundation Models using Interactive Segmentation](https://mlsanity.com/paper/2505.02075/) (2025)
- [FaCT: Faithful Concept Traces for Explaining Neural Network Decisions](https://mlsanity.com/paper/2510.25512/) (2025)
- [Splat and Distill: Augmenting Teachers with Feed-Forward 3D Reconstruction For 3D-Aware Distillation](https://mlsanity.com/paper/2602.06032/) (2026)
- [Cloud4D: Estimating Cloud Properties at a High Spatial and Temporal Resolution](https://mlsanity.com/paper/2511.19431/) (2025)
- [Spectral Probing of Feature Upsamplers in 2D-to-3D Scene Reconstruction](https://mlsanity.com/paper/2603.05787/) (2026)
- [DiveUp: Learning Feature Upsampling from Diverse Vision Foundation Models](https://mlsanity.com/paper/2603.13571/) (2026)
- [PR-IQA: Partial-Reference Image Quality Assessment for Diffusion-Based Novel View Synthesis](https://mlsanity.com/paper/2604.04576/) (2026)
- [MatRes: Zero-Shot Test-Time Model Adaptation for Simultaneous Matching and Restoration](https://mlsanity.com/paper/2604.10081/) (2026)
- [Featurising Pixels from Dynamic 3D Scenes with Linear In-Context Learners](https://mlsanity.com/paper/2604.26488/) (2026)
- [Weighted Reverse Convolution for Feature Upsampling](https://mlsanity.com/paper/2605.17472/) (2026)
- [ViT-Up: Faithful Feature Upsampling for Vision Transformers](https://mlsanity.com/paper/2606.14024/) (2026)
- [RaysUp: Ultra-light Universal Feature Upsampling via Geometry-Aware Ray Representation](https://mlsanity.com/paper/2606.22749/) (2026)
- [From Reconstruction to Decision: A Post-Encoder Plug-in Adapter for Curvilinear Segmentation](https://mlsanity.com/paper/2606.23486/) (2026)
- [SigLIP-HD by Fine-to-Coarse Supervision](https://mlsanity.com/paper/2607.09488/) (2026)
- [Vernata: Self-Supervised Learning of LiDAR Point Representations](https://mlsanity.com/paper/2608.06919/) (2026)

## Abstract (arXiv)

Vision foundation models (VFMs) such as DINOv2 and CLIP have achieved impressive results on various downstream tasks, but their limited feature resolution hampers performance in applications requiring pixel-level understanding. Feature upsampling offers a promising direction to address this challenge. In this work, we identify two critical factors for enhancing feature upsampling: the upsampler architecture and the training objective. For the upsampler architecture, we introduce a coordinate-based cross-attention transformer that integrates the high-resolution images with coordinates and low-resolution VFM features to generate sharp, high-quality features. For the training objective, we propose constructing high-resolution pseudo-groundtruth features by leveraging class-agnostic masks and self-distillation. Our approach effectively captures fine-grained details and adapts flexibly to various input and feature resolutions. Through experiments, we demonstrate that our approach significantly outperforms existing feature upsampling techniques across various downstream tasks. Our code is released at https://github.com/andrehuang/loftup.