# An Empirical Study of Spatial Attention Mechanisms in Deep Networks

arXiv: https://arxiv.org/abs/1904.05873 · HTML: https://mlsanity.com/paper/1904.05873/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Apr 2019.

## Built on by (most cited first, 30 of 58)

- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions](https://mlsanity.com/paper/2211.05778/) (2022)
- [Dynamic Neural Networks: A Survey](https://mlsanity.com/paper/2102.04906/) (2021)
- [Attention Mechanisms in Computer Vision: A Survey](https://mlsanity.com/paper/2111.07624/) (2021)
- [Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs](https://mlsanity.com/paper/2203.06717/) (2022)
- [Involution: Inverting the Inherence of Convolution for Visual Recognition](https://mlsanity.com/paper/2103.06255/) (2021)
- [GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents](https://mlsanity.com/paper/2506.03143/) (2025)
- [Learning Position and Target Consistency for Memory-based Video Object Segmentation](https://mlsanity.com/paper/2104.04329/) (2021)
- [CMT-DeepLab: Clustering Mask Transformers for Panoptic Segmentation](https://mlsanity.com/paper/2206.08948/) (2022)
- [CAVER: Cross-Modal View-Mixed Transformer for Bi-Modal Salient Object Detection](https://mlsanity.com/paper/2112.02363/) (2021)
- [Dual-Awareness Attention for Few-Shot Object Detection](https://mlsanity.com/paper/2102.12152/) (2021)
- [Learned Queries for Efficient Local Attention](https://mlsanity.com/paper/2112.11435/) (2021)
- [Graph Neural Network and Spatiotemporal Transformer Attention for 3D Video Object Detection from Point Clouds](https://mlsanity.com/paper/2207.12659/) (2022)
- [A Unified Efficient Pyramid Transformer for Semantic Segmentation](https://mlsanity.com/paper/2107.14209/) (2021)
- [Learning to Aggregate Multi-Scale Context for Instance Segmentation in Remote Sensing Images](https://mlsanity.com/paper/2111.11057/) (2021)
- [kMaX-DeepLab: k-means Mask Transformer](https://mlsanity.com/paper/2207.04044/) (2022)
- [VIPriors 1: Visual Inductive Priors for Data-Efficient Deep Learning Challenges](https://mlsanity.com/paper/2103.03768/) (2021)
- [Deep Image-based Illumination Harmonization](https://mlsanity.com/paper/2108.00150/) (2021)
- [Recurrent Affine Transformation for Text-to-image Synthesis](https://mlsanity.com/paper/2204.10482/) (2022)
- [Deep Triplet Hashing Network for Case-based Medical Image Retrieval](https://mlsanity.com/paper/2101.12346/) (2021)
- [Adaptive Visual Imitation Learning for Robotic Assisted Feeding Across Varied Bowl Configurations and Food Types](https://mlsanity.com/paper/2403.12891/) (2024)
- [Scaling Up Your Kernels: Large Kernel Design in ConvNets towards Universal Representations](https://mlsanity.com/paper/2410.08049/) (2024)
- [FA-GAN: Fused Attentive Generative Adversarial Networks for MRI Image Super-Resolution](https://mlsanity.com/paper/2108.03920/) (2021)
- [DEER: Detection-agnostic End-to-End Recognizer for Scene Text Spotting](https://mlsanity.com/paper/2203.05122/) (2022)
- [Avatar Knowledge Distillation: Self-ensemble Teacher Paradigm with Uncertainty](https://mlsanity.com/paper/2305.02722/) (2023)
- [EGVD: Event-Guided Video Diffusion Model for Physically Realistic Large-Motion Frame Interpolation](https://mlsanity.com/paper/2503.20268/) (2025)
- [AdaCluster: Adaptive Query-Key Clustering for Sparse Attention in Video Generation](https://mlsanity.com/paper/2604.18348/) (2026)
- [Advancing Autonomous Driving: DepthSense with Radar and Spatial Attention](https://mlsanity.com/paper/2109.05265/) (2021)
- [CPPE-5: Medical Personal Protective Equipment Dataset](https://mlsanity.com/paper/2112.09569/) (2021)
- [Balanced Classification: A Unified Framework for Long-Tailed Object Detection](https://mlsanity.com/paper/2308.02213/) (2023)

## Abstract (arXiv)

Attention mechanisms have become a popular component in deep neural networks, yet there has been little examination of how different influencing factors and methods for computing attention from these factors affect performance. Toward a better general understanding of attention mechanisms, we present an empirical study that ablates various spatial attention elements within a generalized attention formulation, encompassing the dominant Transformer attention as well as the prevalent deformable convolution and dynamic convolution modules. Conducted on a variety of applications, the study yields significant findings about spatial attention in deep networks, some of which run counter to conventional understanding. For example, we find that the query and key content comparison in Transformer attention is negligible for self-attention, but vital for encoder-decoder attention. A proper combination of deformable convolution with key content only saliency achieves the best accuracy-efficiency tradeoff in self-attention. Our results suggest that there exists much room for improvement in the design of attention mechanisms.