# Global Self-Attention Networks for Image Recognition

arXiv: https://arxiv.org/abs/2010.03019 · HTML: https://mlsanity.com/paper/2010.03019/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Oct 2020.

## Built on by (most cited first, 16 of 16)

- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference](https://mlsanity.com/paper/2104.01136/) (2021)
- [Bottleneck Transformers for Visual Recognition](https://mlsanity.com/paper/2101.11605/) (2021)
- [Revisiting ResNets: Improved Training and Scaling Strategies](https://mlsanity.com/paper/2103.07579/) (2021)
- [Generative Adversarial Transformers](https://mlsanity.com/paper/2103.01209/) (2021)
- [LambdaNetworks: Modeling Long-Range Interactions Without Attention](https://mlsanity.com/paper/2102.08602/) (2021)
- [Augmenting Convolutional networks with attention-based aggregation](https://mlsanity.com/paper/2112.13692/) (2021)
- [Interpretable CNN-Multilevel Attention Transformer for Rapid Recognition of Pneumonia from Chest X-Ray Images](https://mlsanity.com/paper/2210.16584/) (2022)
- [Attention and Self-Attention in Random Forests](https://mlsanity.com/paper/2207.04293/) (2022)
- [Multi-Branch with Attention Network for Hand-Based Person Recognition](https://mlsanity.com/paper/2108.02234/) (2021)
- [Local-Aware Global Attention Network for Person Re-Identification Based on Body and Hand Images](https://mlsanity.com/paper/2209.04821/) (2022)
- [CA-Stream: Attention-based pooling for interpretable image recognition](https://mlsanity.com/paper/2404.14996/) (2024)
- [MDD-Net: Multimodal Depression Detection through Mutual Transformer](https://mlsanity.com/paper/2508.08093/) (2025)
- [A Comparison for Anti-noise Robustness of Deep Learning Classification Methods on a Tiny Object Image Dataset: from Convolutional Neural Network to Visual Transformer and Performer](https://mlsanity.com/paper/2106.01927/) (2021)
- [Rethinking Query-Key Pairwise Interactions in Vision Transformers](https://mlsanity.com/paper/2207.00188/) (2022)
- [Ensemble-Based Survival Models with the Self-Attended Beran Estimator Predictions](https://mlsanity.com/paper/2506.07933/) (2025)

## Abstract (arXiv)

Recently, a series of works in computer vision have shown promising results on various image and video understanding tasks using self-attention. However, due to the quadratic computational and memory complexities of self-attention, these works either apply attention only to low-resolution feature maps in later stages of a deep network or restrict the receptive field of attention in each layer to a small local region. To overcome these limitations, this work introduces a new global self-attention module, referred to as the GSA module, which is efficient enough to serve as the backbone component of a deep network. This module consists of two parallel layers: a content attention layer that attends to pixels based only on their content and a positional attention layer that attends to pixels based on their spatial locations. The output of this module is the sum of the outputs of the two layers. Based on the proposed GSA module, we introduce new standalone global attention-based deep networks that use GSA modules instead of convolutions to model pixel interactions. Due to the global extent of the proposed GSA module, a GSA network has the ability to model long-range pixel interactions throughout the network. Our experimental results show that GSA networks outperform the corresponding convolution-based networks significantly on the CIFAR-100 and ImageNet datasets while using less parameters and computations. The proposed GSA networks also outperform various existing attention-based networks on the ImageNet dataset.