# Learning to Upsample by Learning to Sample

arXiv: https://arxiv.org/abs/2308.15085 · HTML: https://mlsanity.com/paper/2308.15085/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 46 references that are in the map.
- First posted on arXiv 29 Aug 2023.

## Builds on (most cited first, 30 of 38)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Deformable ConvNets v2: More Deformable, Better Results](https://mlsanity.com/paper/1811.11168/) (2018)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)
- [AdaBins: Depth Estimation using Adaptive Bins](https://mlsanity.com/paper/2011.14141/) (2020)
- [From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation](https://mlsanity.com/paper/1907.10326/) (2019)
- [Dynamic Filter Networks](https://mlsanity.com/paper/1605.09673/) (2016)
- [Deep Back-Projection Networks For Super-Resolution](https://mlsanity.com/paper/1803.02735/) (2018)
- [Boundary IoU: Improving Object-Centric Image Segmentation Evaluation](https://mlsanity.com/paper/2103.16562/) (2021)
- [Instances as Queries](https://mlsanity.com/paper/2105.01928/) (2021)
- [CARAFE: Content-Aware ReAssembly of FEatures](https://mlsanity.com/paper/1905.02188/) (2019)
- [DepthFormer: Exploiting Long-Range Correlation and Local Information for Accurate Monocular Depth Estimation](https://mlsanity.com/paper/2203.14211/) (2022)

## Built on by (most cited first, 30 of 57)

- [Frequency-aware Feature Fusion for Dense Image Prediction](https://mlsanity.com/paper/2408.12879/) (2024)
- [AnyUp: Universal Feature Upsampling](https://mlsanity.com/paper/2510.12764/) (2025)
- [SegEarth-OV: Towards Training-Free Open-Vocabulary Segmentation for Remote Sensing Images](https://mlsanity.com/paper/2410.01768/) (2024)
- [Playing to Vision Foundation Model's Strengths in Stereo Matching](https://mlsanity.com/paper/2404.06261/) (2024)
- [JAFAR: Jack up Any Feature at Any Resolution](https://mlsanity.com/paper/2506.11136/) (2025)
- [SCSegamba: Lightweight Structure-Aware Vision Mamba for Crack Segmentation in Structures](https://mlsanity.com/paper/2503.01113/) (2025)
- [Unlocking the Potential of MLLMs in Referring Expression Segmentation via a Light-weight Mask Decoder](https://mlsanity.com/paper/2508.04107/) (2025)
- [Annotation-Free Open-Vocabulary Segmentation for Remote-Sensing Images](https://mlsanity.com/paper/2508.18067/) (2025)
- [A Refreshed Similarity-based Upsampler for Direct High-Ratio Feature Upsampling](https://mlsanity.com/paper/2407.02283/) (2024)
- [CCi-YOLOv8n: Enhanced Fire Detection with CARAFE and Context-Guided Modules](https://mlsanity.com/paper/2411.11011/) (2024)
- [NAF: Zero-Shot Feature Upsampling via Neighborhood Attention Filtering](https://mlsanity.com/paper/2511.18452/) (2025)
- [DenseSR: Image Shadow Removal as Dense Prediction](https://mlsanity.com/paper/2507.16472/) (2025)
- [AnyDepth: Depth Estimation Made Easy](https://mlsanity.com/paper/2601.02760/) (2026)
- [XS-VID: A Large-Scale Benchmark for Small Object Detection and Tracking in Videos](https://mlsanity.com/paper/2407.18137/) (2024)
- [FreqU-FNet: Frequency-Aware U-Net for Imbalanced Medical Image Segmentation](https://mlsanity.com/paper/2505.17544/) (2025)
- [Spatial Frequency Modulation for Semantic Segmentation](https://mlsanity.com/paper/2507.11893/) (2025)
- [Paying more attention to local contrast: improving infrared small target detection performance via prior knowledge](https://mlsanity.com/paper/2411.13260/) (2024)
- [C3Net: Context-Contrast Network for Camouflaged Object Detection](https://mlsanity.com/paper/2511.12627/) (2025)
- [UPLiFT: Efficient Pixel-Dense Feature Upsampling with Local Attenders](https://mlsanity.com/paper/2601.17950/) (2026)
- [Flowers: A Warp Drive for Neural PDE Solvers](https://mlsanity.com/paper/2603.04430/) (2026)
- [ARTA: Adaptive Mixed-Resolution Token Allocation for Efficient Dense Feature Extraction](https://mlsanity.com/paper/2603.26258/) (2026)
- [DASSF: Dynamic-Attention Scale-Sequence Fusion for Aerial Object Detection](https://mlsanity.com/paper/2406.12285/) (2024)
- [Implicit Grid Convolution for Multi-Scale Image Super-Resolution](https://mlsanity.com/paper/2408.09674/) (2024)
- [GatedUniPose: A Novel Approach for Pose Estimation Combining UniRepLKNet and Gated Convolution](https://mlsanity.com/paper/2409.07752/) (2024)
- [Beyond Skip Connection: Pooling and Unpooling Design for Elimination Singularities](https://mlsanity.com/paper/2409.13154/) (2024)
- [Dynamic Snake Upsampling Operater and Boundary-Skeleton Weighted Loss for Tubular Structure Segmentation](https://mlsanity.com/paper/2505.08525/) (2025)
- [First Place Solution to the MLCAS 2025 GWFSS Challenge: The Devil is in the Detail and Minority](https://mlsanity.com/paper/2508.17305/) (2025)
- [ChessMamba: Structure-Aware Interleaving of State Spaces for Change Detection in Remote Sensing Images](https://mlsanity.com/paper/2511.19882/) (2025)
- [GateAttentionPose: Enhancing Pose Estimation with Agent Attention and Improved Gated Convolutions](https://mlsanity.com/paper/2409.07798/) (2024)
- [Lighten CARAFE: Dynamic Lightweight Upsampling with Guided Reassemble Kernels](https://mlsanity.com/paper/2410.22139/) (2024)

## Abstract (arXiv)

We present DySample, an ultra-lightweight and effective dynamic upsampler. While impressive performance gains have been witnessed from recent kernel-based dynamic upsamplers such as CARAFE, FADE, and SAPA, they introduce much workload, mostly due to the time-consuming dynamic convolution and the additional sub-network used to generate dynamic kernels. Further, the need for high-res feature guidance of FADE and SAPA somehow limits their application scenarios. To address these concerns, we bypass dynamic convolution and formulate upsampling from the perspective of point sampling, which is more resource-efficient and can be easily implemented with the standard built-in function in PyTorch. We first showcase a naive design, and then demonstrate how to strengthen its upsampling behavior step by step towards our new upsampler, DySample. Compared with former kernel-based dynamic upsamplers, DySample requires no customized CUDA package and has much fewer parameters, FLOPs, GPU memory, and latency. Besides the light-weight characteristics, DySample outperforms other upsamplers across five dense prediction tasks, including semantic segmentation, object detection, instance segmentation, panoptic segmentation, and monocular depth estimation. Code is available at https://github.com/tiny-smart/dysample.