# Deepening Neural Networks Implicitly and Locally via Recurrent Attention Strategy

arXiv: https://arxiv.org/abs/2210.15676 · HTML: https://mlsanity.com/paper/2210.15676/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 21 references that are in the map.
- First posted on arXiv 27 Oct 2022.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks](https://mlsanity.com/paper/1910.03151/) (2019)
- [Attention Mechanisms in Computer Vision: A Survey](https://mlsanity.com/paper/2111.07624/) (2021)
- [MiniViT: Compressing Vision Transformers with Weight Multiplexing](https://mlsanity.com/paper/2204.07154/) (2022)
- [Pad'e Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networks](https://mlsanity.com/paper/1907.06732/) (2019)
- [SA-Net: Shuffle Attention for Deep Convolutional Neural Networks](https://mlsanity.com/paper/2102.00240/) (2021)
- [DIANet: Dense-and-Implicit Attention Network](https://mlsanity.com/paper/1905.10671/) (2019)
- [Visual Attention Methods in Deep Learning: An In-Depth Survey](https://mlsanity.com/paper/2204.07756/) (2022)
- [Instance Enhancement Batch Normalization: an Adaptive Regulator of Batch Noise](https://mlsanity.com/paper/1908.04008/) (2019)
- [Sliced Recursive Transformer](https://mlsanity.com/paper/2111.05297/) (2021)
- [Deep Neural Networks using a Single Neuron: Folded-in-Time Architecture using Feedback-Modulated Delay Loops](https://mlsanity.com/paper/2011.10115/) (2020)
- [Efficient Attention Network: Accelerate Attention by Searching Where to Plug](https://mlsanity.com/paper/2011.14058/) (2020)
- [Compact and Optimal Deep Learning with Recurrent Parameter Generators](https://mlsanity.com/paper/2107.07110/) (2021)
- [Switchable Self-attention Module](https://mlsanity.com/paper/2209.05680/) (2022)

## Abstract (arXiv)

More and more empirical and theoretical evidence shows that deepening neural networks can effectively improve their performance under suitable training settings. However, deepening the backbone of neural networks will inevitably and significantly increase computation and parameter size. To mitigate these problems, we propose a simple-yet-effective Recurrent Attention Strategy (RAS), which implicitly increases the depth of neural networks with lightweight attention modules by local parameter sharing. The extensive experiments on three widely-used benchmark datasets demonstrate that RAS can improve the performance of neural networks at a slight addition of parameter size and computation, performing favorably against other existing well-known attention modules.