# Switchable Self-attention Module

arXiv: https://arxiv.org/abs/2209.05680 · HTML: https://mlsanity.com/paper/2209.05680/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 36 references that are in the map.
- First posted on arXiv 13 Sep 2022.

## Builds on (most cited first, 24 of 24)

- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Image Super-Resolution Using Very Deep Residual Channel Attention Networks](https://mlsanity.com/paper/1807.02758/) (2018)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks](https://mlsanity.com/paper/1910.03151/) (2019)
- [Harmonious Attention Network for Person Re-Identification](https://mlsanity.com/paper/1802.08122/) (2018)
- [DRAW: A Recurrent Neural Network For Image Generation](https://mlsanity.com/paper/1502.04623/) (2015)
- [FcaNet: Frequency Channel Attention Networks](https://mlsanity.com/paper/2012.11879/) (2020)
- [BAM: Bottleneck Attention Module](https://mlsanity.com/paper/1807.06514/) (2018)
- [OCNet: Object Context Network for Scene Parsing](https://mlsanity.com/paper/1809.00916/) (2018)
- [On the Integration of Self-Attention and Convolution](https://mlsanity.com/paper/2111.14556/) (2021)
- [Global Second-order Pooling Convolutional Networks](https://mlsanity.com/paper/1811.12006/) (2018)
- [SRM : A Style-based Recalibration Module for Convolutional Neural Networks](https://mlsanity.com/paper/1903.10829/) (2019)
- [Neural Aggregation Network for Video Face Recognition](https://mlsanity.com/paper/1603.05474/) (2016)
- [DIANet: Dense-and-Implicit Attention Network](https://mlsanity.com/paper/1905.10671/) (2019)
- [Context-aware Attentional Pooling (CAP) for Fine-grained Visual Classification](https://mlsanity.com/paper/2101.06635/) (2021)
- [Convolution-Weight-Distribution Assumption: Rethinking the Criteria of Channel Pruning](https://mlsanity.com/paper/2004.11627/) (2020)
- [Instance Enhancement Batch Normalization: an Adaptive Regulator of Batch Noise](https://mlsanity.com/paper/1908.04008/) (2019)
- [Blending Pruning Criteria for Convolutional Neural Networks](https://mlsanity.com/paper/2107.05033/) (2021)
- [Efficient Attention Network: Accelerate Attention by Searching Where to Plug](https://mlsanity.com/paper/2011.14058/) (2020)
- [The Lottery Ticket Hypothesis for Self-attention in Convolutional Neural Network](https://mlsanity.com/paper/2207.07858/) (2022)

## Built on by (most cited first, 4 of 4)

- [ScaleLong: Towards More Stable Training of Diffusion Model via Scaling Network Long Skip Connection](https://mlsanity.com/paper/2310.13545/) (2023)
- [Understanding Self-attention Mechanism via Dynamical System Perspective](https://mlsanity.com/paper/2308.09939/) (2023)
- [A Generic Shared Attention Mechanism for Various Backbone Neural Networks](https://mlsanity.com/paper/2210.16101/) (2022)
- [Deepening Neural Networks Implicitly and Locally via Recurrent Attention Strategy](https://mlsanity.com/paper/2210.15676/) (2022)

## Abstract (arXiv)

Attention mechanism has gained great success in vision recognition. Many works are devoted to improving the effectiveness of attention mechanism, which finely design the structure of the attention operator. These works need lots of experiments to pick out the optimal settings when scenarios change, which consumes a lot of time and computational resources. In addition, a neural network often contains many network layers, and most studies often use the same attention module to enhance different network layers, which hinders the further improvement of the performance of the self-attention mechanism. To address the above problems, we propose a self-attention module SEM. Based on the input information of the attention module and alternative attention operators, SEM can automatically decide to select and integrate attention operators to compute attention maps. The effectiveness of SEM is demonstrated by extensive experiments on widely used benchmark datasets and popular self-attention networks.