# Spatially Adaptive Inference with Stochastic Feature Sampling and Interpolation

arXiv: https://arxiv.org/abs/2003.08866 · HTML: https://mlsanity.com/paper/2003.08866/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Mar 2020.

## Built on by (most cited first, 30 of 51)

- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [Dynamic Neural Networks: A Survey](https://mlsanity.com/paper/2102.04906/) (2021)
- [Focal Sparse Convolutional Networks for 3D Object Detection](https://mlsanity.com/paper/2204.12463/) (2022)
- [ConvMAE: Masked Convolution Meets Masked Autoencoders](https://mlsanity.com/paper/2205.03892/) (2022)
- [Adaptive Focus for Efficient Video Recognition](https://mlsanity.com/paper/2105.03245/) (2021)
- [QueryDet: Cascaded Sparse Query for Accelerating High-Resolution Small Object Detection](https://mlsanity.com/paper/2103.09136/) (2021)
- [Towards Accurate Image Coding: Improved Autoregressive Image Generation with Dynamic Vector Quantization](https://mlsanity.com/paper/2305.11718/) (2023)
- [PnP-DETR: Towards Efficient Visual Analysis with Transformers](https://mlsanity.com/paper/2109.07036/) (2021)
- [Adaptive Rotated Convolution for Rotated Object Detection](https://mlsanity.com/paper/2303.07820/) (2023)
- [Glance and Focus Networks for Dynamic Visual Recognition](https://mlsanity.com/paper/2201.03014/) (2022)
- [Adaptive Sparse Convolutional Networks with Global Context Enhancement for Faster Object Detection on Drone Images](https://mlsanity.com/paper/2303.14488/) (2023)
- [AdaFocus V2: End-to-End Training of Spatial Dynamic Networks for Video Recognition](https://mlsanity.com/paper/2112.14238/) (2021)
- [Dynamic MDETR: A Dynamic Multimodal Transformer Decoder for Visual Grounding](https://mlsanity.com/paper/2209.13959/) (2022)
- [UMC: A Unified Bandwidth-efficient and Multi-resolution based Collaborative Perception Framework](https://mlsanity.com/paper/2303.12400/) (2023)
- [Spatial Pruned Sparse Convolution for Efficient 3D Object Detection](https://mlsanity.com/paper/2209.14201/) (2022)
- [Latency-aware Unified Dynamic Networks for Efficient Image Recognition](https://mlsanity.com/paper/2308.15949/) (2023)
- [Latency-aware Spatial-wise Dynamic Networks](https://mlsanity.com/paper/2210.06223/) (2022)
- [Dynamic Grained Encoder for Vision Transformers](https://mlsanity.com/paper/2301.03831/) (2023)
- [Anytime Dense Prediction with Confidence Adaptivity](https://mlsanity.com/paper/2104.00749/) (2021)
- [AdaFocusV3: On Unified Spatial-temporal Dynamic Video Recognition](https://mlsanity.com/paper/2209.13465/) (2022)
- [Feature Separation and Recalibration for Adversarial Robustness](https://mlsanity.com/paper/2303.13846/) (2023)
- [ESOD: Efficient Small Object Detection on High-Resolution Images](https://mlsanity.com/paper/2407.16424/) (2024)
- [Computation-efficient Deep Learning for Computer Vision: A Survey](https://mlsanity.com/paper/2308.13998/) (2023)
- [Shadow-Aware Dynamic Convolution for Shadow Removal](https://mlsanity.com/paper/2205.04908/) (2022)
- [Learning to Zoom and Unzoom](https://mlsanity.com/paper/2303.15390/) (2023)
- [AdaBrowse: Adaptive Video Browser for Efficient Continuous Sign Language Recognition](https://mlsanity.com/paper/2308.08327/) (2023)
- [BlockCopy: High-Resolution Video Processing with Block-Sparse Feature Propagation and Online Policies](https://mlsanity.com/paper/2108.09376/) (2021)
- [ERA: Expert Retrieval and Assembly for Early Action Prediction](https://mlsanity.com/paper/2207.09675/) (2022)
- [Uni-AdaFocus: Spatial-temporal Dynamic Computation for Video Recognition](https://mlsanity.com/paper/2412.11228/) (2024)
- [DS-Net++: Dynamic Weight Slicing for Efficient Inference in CNNs and Transformers](https://mlsanity.com/paper/2109.10060/) (2021)

## Abstract (arXiv)

In the feature maps of CNNs, there commonly exists considerable spatial redundancy that leads to much repetitive processing. Towards reducing this superfluous computation, we propose to compute features only at sparsely sampled locations, which are probabilistically chosen according to activation responses, and then densely reconstruct the feature map with an efficient interpolation procedure. With this sampling-interpolation scheme, our network avoids expending computation on spatial locations that can be effectively interpolated, while being robust to activation prediction errors through broadly distributed sampling. A technical challenge of this sampling-based approach is that the binary decision variables for representing discrete sampling locations are non-differentiable, making them incompatible with backpropagation. To circumvent this issue, we make use of a reparameterization trick based on the Gumbel-Softmax distribution, with which backpropagation can iterate these variables towards binary values. The presented network is experimentally shown to save substantial computation while maintaining accuracy over a variety of computer vision tasks.