# Stochastic Downsampling for Cost-Adjustable Inference and Improved Regularization in Convolutional Networks

arXiv: https://arxiv.org/abs/1801.09335 · HTML: https://mlsanity.com/paper/1801.09335/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jan 2018.

## Built on by (most cited first, 7 of 7)

- [Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications](https://mlsanity.com/paper/2204.11786/) (2022)
- [Multi-Scale Aligned Distillation for Low-Resolution Detection](https://mlsanity.com/paper/2109.06875/) (2021)
- [Supernet Training for Federated Image Classification under System Heterogeneity](https://mlsanity.com/paper/2206.01366/) (2022)
- [Terra: Imperative-Symbolic Co-Execution of Imperative Deep Learning Programs](https://mlsanity.com/paper/2201.09210/) (2022)
- [Scale-Space Hypernetworks for Efficient Biomedical Imaging](https://mlsanity.com/paper/2304.05448/) (2023)
- [Cross-Domain Knowledge Distillation for Low-Resolution Human Pose Estimation](https://mlsanity.com/paper/2405.11448/) (2024)
- [Focus on Low-Resolution Information: Multi-Granular Information-Lossless Model for Low-Resolution Human Pose Estimation](https://mlsanity.com/paper/2405.12247/) (2024)

## Abstract (arXiv)

It is desirable to train convolutional networks (CNNs) to run more efficiently during inference. In many cases however, the computational budget that the system has for inference cannot be known beforehand during training, or the inference budget is dependent on the changing real-time resource availability. Thus, it is inadequate to train just inference-efficient CNNs, whose inference costs are not adjustable and cannot adapt to varied inference budgets. We propose a novel approach for cost-adjustable inference in CNNs - Stochastic Downsampling Point (SDPoint). During training, SDPoint applies feature map downsampling to a random point in the layer hierarchy, with a random downsampling ratio. The different stochastic downsampling configurations known as SDPoint instances (of the same model) have computational costs different from each other, while being trained to minimize the same prediction loss. Sharing network parameters across different instances provides significant regularization boost. During inference, one may handpick a SDPoint instance that best fits the inference budget. The effectiveness of SDPoint, as both a cost-adjustable inference approach and a regularizer, is validated through extensive experiments on image classification.