# SA-Net: Shuffle Attention for Deep Convolutional Neural Networks

arXiv: https://arxiv.org/abs/2102.00240 · HTML: https://mlsanity.com/paper/2102.00240/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 31 references that are in the map.
- First posted on arXiv 30 Jan 2021.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [CCNet: Criss-Cross Attention for Semantic Segmentation](https://mlsanity.com/paper/1811.11721/) (2018)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks](https://mlsanity.com/paper/1910.03151/) (2019)
- [Selective Kernel Networks](https://mlsanity.com/paper/1903.06586/) (2019)
- [GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond](https://mlsanity.com/paper/1904.11492/) (2019)
- [Asymmetric Non-local Neural Networks for Semantic Segmentation](https://mlsanity.com/paper/1908.07678/) (2019)
- [Expectation-Maximization Attention Networks for Semantic Segmentation](https://mlsanity.com/paper/1907.13426/) (2019)
- [OCNet: Object Context Network for Scene Parsing](https://mlsanity.com/paper/1809.00916/) (2018)
- [SRM : A Style-based Recalibration Module for Convolutional Neural Networks](https://mlsanity.com/paper/1903.10829/) (2019)
- [Spatial Group-wise Enhance: Improving Semantic Feature Learning in Convolutional Networks](https://mlsanity.com/paper/1905.09646/) (2019)

## Built on by (most cited first, 30 of 54)

- [Transformers in Medical Imaging: A Survey](https://mlsanity.com/paper/2201.09873/) (2022)
- [ResT: An Efficient Transformer for Visual Recognition](https://mlsanity.com/paper/2105.13677/) (2021)
- [Packed-Ensembles for Efficient Uncertainty Estimation](https://mlsanity.com/paper/2210.09184/) (2022)
- [Efficient Multi-Scale Attention Module with Cross-Spatial Learning](https://mlsanity.com/paper/2305.13563/) (2023)
- [WaterScenes: A Multi-Task 4D Radar-Camera Fusion Dataset and Benchmarks for Autonomous Driving on Water Surfaces](https://mlsanity.com/paper/2307.06505/) (2023)
- [Feature Distillation Interaction Weighting Network for Lightweight Image Super-Resolution](https://mlsanity.com/paper/2112.08655/) (2021)
- [Personalizing Federated Medical Image Segmentation via Local Calibration](https://mlsanity.com/paper/2207.04655/) (2022)
- [Salient Object Detection in Optical Remote Sensing Images Driven by Transformer](https://mlsanity.com/paper/2309.08206/) (2023)
- [Achelous: A Fast Unified Water-surface Panoptic Perception Framework based on Fusion of Monocular Camera and 4D mmWave Radar](https://mlsanity.com/paper/2307.07102/) (2023)
- [EPSANet: An Efficient Pyramid Squeeze Attention Block on Convolutional Neural Network](https://mlsanity.com/paper/2105.14447/) (2021)
- [RGB-X Object Detection via Scene-Specific Fusion Modules](https://mlsanity.com/paper/2310.19372/) (2023)
- [MSDet: Receptive Field Enhanced Multiscale Detection for Tiny Pulmonary Nodule](https://mlsanity.com/paper/2409.14028/) (2024)
- [WaterVG: Waterway Visual Grounding based on Text-Guided Vision and mmWave Radar](https://mlsanity.com/paper/2403.12686/) (2024)
- [ASY-VRNet: Waterway Panoptic Driving Perception Model based on Asymmetric Fair Fusion of Vision and 4D mmWave Radar](https://mlsanity.com/paper/2308.10287/) (2023)
- [YOLOv9 for Fracture Detection in Pediatric Wrist Trauma X-ray Images](https://mlsanity.com/paper/2403.11249/) (2024)
- [SCSA: Exploring the Synergistic Effects Between Spatial and Channel Attention](https://mlsanity.com/paper/2407.05128/) (2024)
- [BA-Net: Bridge Attention for Deep Convolutional Neural Networks](https://mlsanity.com/paper/2112.04150/) (2021)
- [DMSANet: Dual Multi Scale Attention Network](https://mlsanity.com/paper/2106.08382/) (2021)
- [ELA: Efficient Local Attention for Deep Convolutional Neural Networks](https://mlsanity.com/paper/2403.01123/) (2024)
- [YOLOv8-AM: YOLOv8 Based on Effective Attention Mechanisms for Pediatric Wrist Fracture Detection](https://mlsanity.com/paper/2402.09329/) (2024)
- [AIM 2024 Challenge on Video Saliency Prediction: Methods and Results](https://mlsanity.com/paper/2409.14827/) (2024)
- [Physics Inspired Hybrid Attention for SAR Target Recognition](https://mlsanity.com/paper/2309.15697/) (2023)
- [DPNet: Dual-Path Network for Real-time Object Detection with Lightweight Attention](https://mlsanity.com/paper/2209.13933/) (2022)
- [Residual Kolmogorov-Arnold Network for Enhanced Deep Learning](https://mlsanity.com/paper/2410.05500/) (2024)
- [Achelous++: Power-Oriented Water-Surface Panoptic Perception Framework on Edge Devices based on Vision-Radar Fusion and Pruning of Heterogeneous Modalities](https://mlsanity.com/paper/2312.08851/) (2023)
- [Parameter-Free Channel Attention for Image Classification and Super-Resolution](https://mlsanity.com/paper/2303.11055/) (2023)
- [SegT: A Novel Separated Edge-guidance Transformer Network for Polyp Segmentation](https://mlsanity.com/paper/2306.10773/) (2023)
- [Attention Modules Improve Image-Level Anomaly Detection for Industrial Inspection: A DifferNet Case Study](https://mlsanity.com/paper/2311.02747/) (2023)
- [VyAnG-Net: A Novel Multi-Modal Sarcasm Recognition Model by Uncovering Visual, Acoustic and Glossary Features](https://mlsanity.com/paper/2408.10246/) (2024)
- [TreeNet: A lightweight One-Shot Aggregation Convolutional Network](https://mlsanity.com/paper/2109.12342/) (2021)

## Abstract (arXiv)

Attention mechanisms, which enable a neural network to accurately focus on all the relevant elements of the input, have become an essential component to improve the performance of deep neural networks. There are mainly two attention mechanisms widely used in computer vision studies, textitspatial attention and textitchannel attention, which aim to capture the pixel-level pairwise relationship and channel dependency, respectively. Although fusing them together may achieve better performance than their individual implementations, it will inevitably increase the computational overhead. In this paper, we propose an efficient Shuffle Attention (SA) module to address this issue, which adopts Shuffle Units to combine two types of attention mechanisms effectively. Specifically, SA first groups channel dimensions into multiple sub-features before processing them in parallel. Then, for each sub-feature, SA utilizes a Shuffle Unit to depict feature dependencies in both spatial and channel dimensions. After that, all sub-features are aggregated and a "channel shuffle" operator is adopted to enable information communication between different sub-features. The proposed SA module is efficient yet effective, e.g., the parameters and computations of SA against the backbone ResNet50 are 300 vs. 25.56M and 2.76e-3 GFLOPs vs. 4.12 GFLOPs, respectively, and the performance boost is more than 1.34% in terms of Top-1 accuracy. Extensive experimental results on common-used benchmarks, including ImageNet-1k for classification, MS COCO for object detection, and instance segmentation, demonstrate that the proposed SA outperforms the current SOTA methods significantly by achieving higher accuracy while having lower model complexity. The code and models are available at https://github.com/wofmanaf/SA-Net.