# Learning to Zoom: a Saliency-Based Sampling Layer for Neural Networks

arXiv: https://arxiv.org/abs/1809.03355 · HTML: https://mlsanity.com/paper/1809.03355/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Sep 2018.

## Built on by (most cited first, 30 of 56)

- [Dynamic Neural Networks: A Survey](https://mlsanity.com/paper/2102.04906/) (2021)
- [Token Pooling in Vision Transformers](https://mlsanity.com/paper/2110.03860/) (2021)
- [The Way to my Heart is through Contrastive Learning: Remote Photoplethysmography from Unlabelled Video](https://mlsanity.com/paper/2111.09748/) (2021)
- [Glance and Focus Networks for Dynamic Visual Recognition](https://mlsanity.com/paper/2201.03014/) (2022)
- [RAMS-Trans: Recurrent Attention Multi-scale Transformer forFine-grained Image Recognition](https://mlsanity.com/paper/2107.08192/) (2021)
- [ImageNet-Hard: The Hardest Images Remaining from a Study of the Power of Zoom and Spatial Biases in Image Classification](https://mlsanity.com/paper/2304.05538/) (2023)
- [FOVEA: Foveated Image Magnification for Autonomous Navigation](https://mlsanity.com/paper/2108.12102/) (2021)
- [Coarse-Fine Networks for Temporal Activity Detection in Videos](https://mlsanity.com/paper/2103.01302/) (2021)
- [ZoomTrack: Target-aware Non-uniform Resizing for Efficient Visual Tracking](https://mlsanity.com/paper/2310.10071/) (2023)
- [Dynamic Spatial Sparsification for Efficient Vision Transformers and Convolutional Neural Networks](https://mlsanity.com/paper/2207.01580/) (2022)
- [Learning to Downsample for Segmentation of Ultra-High Resolution Images](https://mlsanity.com/paper/2109.11071/) (2021)
- [SIM-Trans: Structure Information Modeling Transformer for Fine-grained Visual Categorization](https://mlsanity.com/paper/2208.14607/) (2022)
- [Adaptive Mutual Supervision for Weakly-Supervised Temporal Action Localization](https://mlsanity.com/paper/2104.02357/) (2021)
- [A free lunch from ViT:Adaptive Attention Multi-scale Fusion Transformer for Fine-grained Visual Recognition](https://mlsanity.com/paper/2110.01240/) (2021)
- [Constructive Distortion: Improving MLLMs with Attention-Guided Image Warping](https://mlsanity.com/paper/2510.09741/) (2025)
- [SALISA: Saliency-based Input Sampling for Efficient Video Object Detection](https://mlsanity.com/paper/2204.02397/) (2022)
- [Learning to Zoom and Unzoom](https://mlsanity.com/paper/2303.15390/) (2023)
- [Zero-Shot Aerial Object Detection with Visual Description Regularization](https://mlsanity.com/paper/2402.18233/) (2024)
- [Task-adaptive Spatial-Temporal Video Sampler for Few-shot Action Recognition](https://mlsanity.com/paper/2207.09759/) (2022)
- [Efficient High-Resolution Deep Learning: A Survey](https://mlsanity.com/paper/2207.13050/) (2022)
- [Foveation in the Era of Deep Learning](https://mlsanity.com/paper/2312.01450/) (2023)
- [Learning Canonical 3D Object Representation for Fine-Grained Recognition](https://mlsanity.com/paper/2108.04628/) (2021)
- [Part-guided Relational Transformers for Fine-grained Visual Recognition](https://mlsanity.com/paper/2212.13685/) (2022)
- [Few-shot Action Recognition via Intra- and Inter-Video Information Maximization](https://mlsanity.com/paper/2305.06114/) (2023)
- [Instance-Warp: Saliency Guided Image Warping for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2403.12712/) (2024)
- [AdaGlimpse: Active Visual Exploration with Arbitrary Glimpse Position and Scale](https://mlsanity.com/paper/2404.03482/) (2024)
- [Convolutional Neural Networks with Gated Recurrent Connections](https://mlsanity.com/paper/2106.02859/) (2021)
- [Learn From Zoom: Decoupled Supervised Contrastive Learning For WCE Image Classification](https://mlsanity.com/paper/2401.05771/) (2024)
- [Content-Adaptive Downsampling in Convolutional Neural Networks](https://mlsanity.com/paper/2305.09504/) (2023)
- [No More Sliding Window: Efficient 3D Medical Image Segmentation with Differentiable Top-k Patch Sampling](https://mlsanity.com/paper/2501.10814/) (2025)

## Abstract (arXiv)

We introduce a saliency-based distortion layer for convolutional neural networks that helps to improve the spatial sampling of input data for a given task. Our differentiable layer can be added as a preprocessing block to existing task networks and trained altogether in an end-to-end fashion. The effect of the layer is to efficiently estimate how to sample from the original data in order to boost task performance. For example, for an image classification task in which the original data might range in size up to several megapixels, but where the desired input images to the task network are much smaller, our layer learns how best to sample from the underlying high resolution data in a manner which preserves task-relevant information better than uniform downsampling. This has the effect of creating distorted, caricature-like intermediate images, in which idiosyncratic elements of the image that improve task performance are zoomed and exaggerated. Unlike alternative approaches such as spatial transformer networks, our proposed layer is inspired by image saliency, computed efficiently from uniformly downsampled data, and degrades gracefully to a uniform sampling strategy under uncertainty. We apply our layer to improve existing networks for the tasks of human gaze estimation and fine-grained object classification. Code for our method is available in: http://github.com/recasens/Saliency-Sampler