# Residual Attention Network for Image Classification

arXiv: https://arxiv.org/abs/1704.06904 · HTML: https://mlsanity.com/paper/1704.06904/

## Key facts

- Cited by 386 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Apr 2017.

## Built on by (most cited first, 30 of 386)

- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference](https://mlsanity.com/paper/2104.01136/) (2021)
- [VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text](https://mlsanity.com/paper/2104.11178/) (2021)
- [Dynamic Neural Networks: A Survey](https://mlsanity.com/paper/2102.04906/) (2021)
- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [CMT: Convolutional Neural Networks Meet Vision Transformers](https://mlsanity.com/paper/2107.06263/) (2021)
- [Escaping the Big Data Paradigm with Compact Transformers](https://mlsanity.com/paper/2104.05704/) (2021)
- [Attention Mechanisms in Computer Vision: A Survey](https://mlsanity.com/paper/2111.07624/) (2021)
- [WebFace260M: A Benchmark Unveiling the Power of Million-Scale Deep Face Recognition](https://mlsanity.com/paper/2103.04098/) (2021)
- [Visual Attention Network](https://mlsanity.com/paper/2202.09741/) (2022)
- [VOLO: Vision Outlooker for Visual Recognition](https://mlsanity.com/paper/2106.13112/) (2021)
- [PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognition](https://mlsanity.com/paper/2403.17695/) (2024)
- [Vision Transformer for Small-Size Datasets](https://mlsanity.com/paper/2112.13492/) (2021)
- [Counterfactual Attention Learning for Fine-Grained Visual Categorization and Re-identification](https://mlsanity.com/paper/2108.08728/) (2021)
- [LViT: Language meets Vision Transformer in Medical Image Segmentation](https://mlsanity.com/paper/2206.14718/) (2022)
- [Attention Spiking Neural Networks](https://mlsanity.com/paper/2209.13929/) (2022)
- [Recent advances and clinical applications of deep learning in medical image analysis](https://mlsanity.com/paper/2105.13381/) (2021)
- [Locate then Segment: A Strong Pipeline for Referring Image Segmentation](https://mlsanity.com/paper/2103.16284/) (2021)
- [Real-Time Scene Text Detection with Differentiable Binarization and Adaptive Scale Fusion](https://mlsanity.com/paper/2202.10304/) (2022)
- [Learning Normal Dynamics in Videos with Meta Prototype Network](https://mlsanity.com/paper/2104.06689/) (2021)
- [Differentiable Patch Selection for Image Recognition](https://mlsanity.com/paper/2104.03059/) (2021)
- [DeepMultiCap: Performance Capture of Multiple Characters Using Sparse Multiview Cameras](https://mlsanity.com/paper/2105.00261/) (2021)
- [Synthetic Data -- what, why and how?](https://mlsanity.com/paper/2205.03257/) (2022)
- [MaskTune: Mitigating Spurious Correlations by Forcing to Explore](https://mlsanity.com/paper/2210.00055/) (2022)
- [Omni-Dimensional Dynamic Convolution](https://mlsanity.com/paper/2209.07947/) (2022)
- [Decoupled Dynamic Filter Networks](https://mlsanity.com/paper/2104.14107/) (2021)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [Adaptive Rotated Convolution for Rotated Object Detection](https://mlsanity.com/paper/2303.07820/) (2023)
- [FaceX-Zoo: A PyTorch Toolbox for Face Recognition](https://mlsanity.com/paper/2101.04407/) (2021)
- [Multi-task Learning with Attention for End-to-end Autonomous Driving](https://mlsanity.com/paper/2104.10753/) (2021)

## Abstract (arXiv)

In this work, we propose "Residual Attention Network", a convolutional neural network using attention mechanism which can incorporate with state-of-art feed forward network architecture in an end-to-end training fashion. Our Residual Attention Network is built by stacking Attention Modules which generate attention-aware features. The attention-aware features from different modules change adaptively as layers going deeper. Inside each Attention Module, bottom-up top-down feedforward structure is used to unfold the feedforward and feedback attention process into a single feedforward process. Importantly, we propose attention residual learning to train very deep Residual Attention Networks which can be easily scaled up to hundreds of layers. Extensive analyses are conducted on CIFAR-10 and CIFAR-100 datasets to verify the effectiveness of every module mentioned above. Our Residual Attention Network achieves state-of-the-art object recognition performance on three benchmark datasets including CIFAR-10 (3.90% error), CIFAR-100 (20.45% error) and ImageNet (4.8% single model and single crop, top-5 error). Note that, our method achieves 0.6% top-1 accuracy improvement with 46% trunk depth and 69% forward FLOPs comparing to ResNet-200. The experiment also demonstrates that our network is robust against noisy labels.