# Salient Positions based Attention Network for Image Classification

arXiv: https://arxiv.org/abs/2106.04996 · HTML: https://mlsanity.com/paper/2106.04996/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 37 references that are in the map.
- First posted on arXiv 9 Jun 2021.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [CCNet: Criss-Cross Attention for Semantic Segmentation](https://mlsanity.com/paper/1811.11721/) (2018)
- [ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks](https://mlsanity.com/paper/1910.03151/) (2019)
- [Selective Kernel Networks](https://mlsanity.com/paper/1903.06586/) (2019)
- [Efficient Attention: Attention with Linear Complexities](https://mlsanity.com/paper/1812.01243/) (2018)
- [GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond](https://mlsanity.com/paper/1904.11492/) (2019)
- [Stacked Attention Networks for Image Question Answering](https://mlsanity.com/paper/1511.02274/) (2015)
- [SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioning](https://mlsanity.com/paper/1611.05594/) (2016)
- [Local Relation Networks for Image Recognition](https://mlsanity.com/paper/1904.11491/) (2019)
- [Salient Object Detection in the Deep Learning Era: An In-Depth Survey](https://mlsanity.com/paper/1904.09146/) (2019)
- [Expectation-Maximization Attention Networks for Semantic Segmentation](https://mlsanity.com/paper/1907.13426/) (2019)
- [Gather-Excite: Exploiting Feature Context in Convolutional Neural Networks](https://mlsanity.com/paper/1810.12348/) (2018)
- [A2-Nets: Double Attention Networks](https://mlsanity.com/paper/1810.11579/) (2018)
- [Recalibrating Fully Convolutional Networks with Spatial and Channel 'Squeeze & Excitation' Blocks](https://mlsanity.com/paper/1808.08127/) (2018)
- [Attentive Group Equivariant Convolutional Networks](https://mlsanity.com/paper/2002.03830/) (2020)
- [An Empirical Study of Spatial Attention Mechanisms in Deep Networks](https://mlsanity.com/paper/1904.05873/) (2019)
- [Interlaced Sparse Self-Attention for Semantic Segmentation](https://mlsanity.com/paper/1907.12273/) (2019)
- [Neural Architecture Search for Lightweight Non-Local Networks](https://mlsanity.com/paper/2004.01961/) (2020)
- [DenseNet Models for Tiny ImageNet Classification](https://mlsanity.com/paper/1904.10429/) (2019)

## Built on by (most cited first, 1 of 1)

- [Spatial Bias for Attention-free Non-local Neural Networks](https://mlsanity.com/paper/2302.12505/) (2023)

## Abstract (arXiv)

The self-attention mechanism has attracted wide publicity for its most important advantage of modeling long dependency, and its variations in computer vision tasks, the non-local block tries to model the global dependency of the input feature maps. Gathering global contextual information will inevitably need a tremendous amount of memory and computing resources, which has been extensively studied in the past several years. However, there is a further problem with the self-attention scheme: is all information gathered from the global scope helpful for the contextual modelling? To our knowledge, few studies have focused on the problem. Aimed at both questions this paper proposes the salient positions-based attention scheme SPANet, which is inspired by some interesting observations on the attention maps and affinity matrices generated in self-attention scheme. We believe these observations are beneficial for better understanding of the self-attention. SPANet uses the salient positions selection algorithm to select only a limited amount of salient points to attend in the attention map computing. This approach will not only spare a lot of memory and computing resources, but also try to distill the positive information from the transformation of the input feature maps. In the implementation, considering the feature maps with channel high dimensions, which are completely different from the general visual image, we take the squared power of the feature maps along the channel dimension as the saliency metric of the positions. In general, different from the non-local block method, SPANet models the contextual information using only the selected positions instead of all, along the channel dimension instead of space dimension. Our source code is available at https://github.com/likyoo/SPANet.