# Efficient Attention Network: Accelerate Attention by Searching Where to Plug

arXiv: https://arxiv.org/abs/2011.14058 · HTML: https://mlsanity.com/paper/2011.14058/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Nov 2020.

## Built on by (most cited first, 9 of 9)

- [Blending Pruning Criteria for Convolutional Neural Networks](https://mlsanity.com/paper/2107.05033/) (2021)
- [Understanding Self-attention Mechanism via Dynamical System Perspective](https://mlsanity.com/paper/2308.09939/) (2023)
- [AlterSGD: Finding Flat Minima for Continual Learning by Alternative Training](https://mlsanity.com/paper/2107.05804/) (2021)
- [A Generic Shared Attention Mechanism for Various Backbone Neural Networks](https://mlsanity.com/paper/2210.16101/) (2022)
- [Switchable Self-attention Module](https://mlsanity.com/paper/2209.05680/) (2022)
- [LSAS: Lightweight Sub-attention Strategy for Alleviating Attention Bias Problem](https://mlsanity.com/paper/2305.05200/) (2023)
- [Mix-Pooling Strategy for Attention Mechanism](https://mlsanity.com/paper/2208.10322/) (2022)
- [Causal Inference for Chatting Handoff](https://mlsanity.com/paper/2210.02862/) (2022)
- [Deepening Neural Networks Implicitly and Locally via Recurrent Attention Strategy](https://mlsanity.com/paper/2210.15676/) (2022)

## Abstract (arXiv)

Recently, many plug-and-play self-attention modules are proposed to enhance the model generalization by exploiting the internal information of deep convolutional neural networks (CNNs). Previous works lay an emphasis on the design of attention module for specific functionality, e.g., light-weighted or task-oriented attention. However, they ignore the importance of where to plug in the attention module since they connect the modules individually with each block of the entire CNN backbone for granted, leading to incremental computational cost and number of parameters with the growth of network depth. Thus, we propose a framework called Efficient Attention Network (EAN) to improve the efficiency for the existing attention modules. In EAN, we leverage the sharing mechanism (Huang et al. 2020) to share the attention module within the backbone and search where to connect the shared attention module via reinforcement learning. Finally, we obtain the attention network with sparse connections between the backbone and modules, while (1) maintaining accuracy (2) reducing extra parameter increment and (3) accelerating inference. Extensive experiments on widely-used benchmarks and popular attention networks show the effectiveness of EAN. Furthermore, we empirically illustrate that our EAN has the capacity of transferring to other tasks and capturing the informative features. The code is available at https://github.com/gbup-group/EAN-efficient-attention-network.