# Shift-and-Balance Attention

arXiv: https://arxiv.org/abs/2103.13080 · HTML: https://mlsanity.com/paper/2103.13080/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 37 of its 49 references that are in the map.
- First posted on arXiv 24 Mar 2021.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [CCNet: Criss-Cross Attention for Semantic Segmentation](https://mlsanity.com/paper/1811.11721/) (2018)
- [ResNeSt: Split-Attention Networks](https://mlsanity.com/paper/2004.08955/) (2020)
- [Selective Kernel Networks](https://mlsanity.com/paper/1903.06586/) (2019)
- [Residual Attention Network for Image Classification](https://mlsanity.com/paper/1704.06904/) (2017)
- [Dynamic Convolution: Attention over Convolution Kernels](https://mlsanity.com/paper/1912.03458/) (2019)
- [SkipNet: Learning Dynamic Routing in Convolutional Networks](https://mlsanity.com/paper/1711.09485/) (2017)
- [Dynamic Filter Networks](https://mlsanity.com/paper/1605.09673/) (2016)
- [A Structured Self-attentive Sentence Embedding](https://mlsanity.com/paper/1703.03130/) (2017)
- [Slimmable Neural Networks](https://mlsanity.com/paper/1812.08928/) (2018)
- [CondConv: Conditionally Parameterized Convolutions for Efficient Inference](https://mlsanity.com/paper/1904.04971/) (2019)
- [BlockDrop: Dynamic Inference Paths in Residual Networks](https://mlsanity.com/paper/1711.08393/) (2017)
- [BAM: Bottleneck Attention Module](https://mlsanity.com/paper/1807.06514/) (2018)

## Abstract (arXiv)

Attention is an effective mechanism to improve the deep model capability. Squeeze-and-Excite (SE) introduces a light-weight attention branch to enhance the network's representational power. The attention branch is gated using the Sigmoid function and multiplied by the feature map's trunk branch. It is too sensitive to coordinate and balance the trunk and attention branches' contributions. To control the attention branch's influence, we propose a new attention method, called Shift-and-Balance (SB). Different from Squeeze-and-Excite, the attention branch is regulated by the learned control factor to control the balance, then added into the feature map's trunk branch. Experiments show that Shift-and-Balance attention significantly improves the accuracy compared to Squeeze-and-Excite when applied in more layers, increasing more size and capacity of a network. Moreover, Shift-and-Balance attention achieves better or close accuracy compared to the state-of-art Dynamic Convolution.