# Hard-Attention for Scalable Image Classification

arXiv: https://arxiv.org/abs/2102.10212 · HTML: https://mlsanity.com/paper/2102.10212/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 65 of its 88 references that are in the map.
- First posted on arXiv 20 Feb 2021.

## Builds on (most cited first, 30 of 65)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)

## Built on by (most cited first, 18 of 18)

- [Efficient Classification of Very Large Images with Tiny Objects](https://mlsanity.com/paper/2106.02694/) (2021)
- [A Probabilistic Hard Attention Model For Sequentially Observed Scenes](https://mlsanity.com/paper/2111.07534/) (2021)
- [Consistency driven Sequential Transformers Attention Model for Partially Observable Scenes](https://mlsanity.com/paper/2204.00656/) (2022)
- [A Visual Active Search Framework for Geospatial Exploration](https://mlsanity.com/paper/2211.15788/) (2022)
- [A Partially Supervised Reinforcement Learning Framework for Visual Active Search](https://mlsanity.com/paper/2310.09689/) (2023)
- [AdaGlimpse: Active Visual Exploration with Arbitrary Glimpse Position and Scale](https://mlsanity.com/paper/2404.03482/) (2024)
- [GQHAN: A Grover-inspired Quantum Hard Attention Network](https://mlsanity.com/paper/2401.14089/) (2024)
- [Recurrent Attention Models with Object-centric Capsule Representation for Multi-object Recognition](https://mlsanity.com/paper/2110.04954/) (2021)
- [Patch Gradient Descent: Training Neural Networks on Very Large Images](https://mlsanity.com/paper/2301.13817/) (2023)
- [GliTr: Glimpse Transformers with Spatiotemporal Consistency for Online Action Prediction](https://mlsanity.com/paper/2210.13605/) (2022)
- [Predictive Temporal Attention on Event-based Video Stream for Energy-efficient Situation Awareness](https://mlsanity.com/paper/2402.08936/) (2024)
- [CanViT: Toward Active-Vision Foundation Models](https://mlsanity.com/paper/2603.22570/) (2026)
- [Learning Downstream Task by Selectively Capturing Complementary Knowledge from Multiple Self-supervisedly Learning Pretexts](https://mlsanity.com/paper/2204.05248/) (2022)
- [Harnessing Deep Learning and Satellite Imagery for Post-Buyout Land Cover Mapping](https://mlsanity.com/paper/2401.07500/) (2024)
- [Resource Efficient Perception for Vision Systems](https://mlsanity.com/paper/2405.07166/) (2024)
- [From RISC-V Cores to Neuromorphic Arrays: A Tutorial on Building Scalable Digital Neuromorphic Processors](https://mlsanity.com/paper/2512.00113/) (2025)
- [CPR: Chained Perceptual Refinement for Coarse-to-Fine Medical Image Classification](https://mlsanity.com/paper/2607.02591/) (2026)
- [Exploring Budgeted Image Classification with Content-Sensitive Resource Allocation](https://mlsanity.com/paper/2607.23997/) (2026)

## Abstract (arXiv)

Can we leverage high-resolution information without the unsustainable quadratic complexity to input scale? We propose Traversal Network (TNet), a novel multi-scale hard-attention architecture, which traverses image scale-space in a top-down fashion, visiting only the most informative image regions along the way. TNet offers an adjustable trade-off between accuracy and complexity, by changing the number of attended image locations. We compare our model against hard-attention baselines on ImageNet, achieving higher accuracy with less resources (FLOPs, processing time and memory). We further test our model on fMoW dataset, where we process satellite images of size up to 896 × 896 px, getting up to 2.5x faster processing compared to baselines operating on the same resolution, while achieving higher accuracy as well. TNet is modular, meaning that most classification models could be adopted as its backbone for feature extraction, making the reported performance gains orthogonal to benefits offered by existing optimized deep models. Finally, hard-attention guarantees a degree of interpretability to our model's predictions, without any extra cost beyond inference. Code is available at hrefhttps://github.com/Tpap/TNetgithub.com/Tpap/TNet.