# Visual Attention Methods in Deep Learning: An In-Depth Survey

arXiv: https://arxiv.org/abs/2204.07756 · HTML: https://mlsanity.com/paper/2204.07756/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 139 of its 194 references that are in the map.
- First posted on arXiv 16 Apr 2022.

## Builds on (most cited first, 30 of 139)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)

## Built on by (most cited first, 30 of 37)

- [Rethinking Mobile Block for Efficient Attention-based Models](https://mlsanity.com/paper/2301.01146/) (2023)
- [Artificial General Intelligence for Medical Imaging Analysis](https://mlsanity.com/paper/2306.05480/) (2023)
- [A Survey on Personalized Content Synthesis with Diffusion Models](https://mlsanity.com/paper/2405.05538/) (2024)
- [EXMOS: Explanatory Model Steering Through Multifaceted Explanations and Data Configurations](https://mlsanity.com/paper/2402.00491/) (2024)
- [FactorizePhys: Matrix Factorization for Multidimensional Attention in Remote Physiological Sensing](https://mlsanity.com/paper/2411.01542/) (2024)
- [Reasoning-Enhanced Object-Centric Learning for Videos](https://mlsanity.com/paper/2403.15245/) (2024)
- [A Theoretical Insight into Attack and Defense of Gradient Leakage in Transformer](https://mlsanity.com/paper/2311.13624/) (2023)
- [EMOv2: Pushing 5M Vision Model Frontier](https://mlsanity.com/paper/2412.06674/) (2024)
- [SlotPi: Physics-informed Object-centric Reasoning Models](https://mlsanity.com/paper/2506.10778/) (2025)
- [Attention and Self-Attention in Random Forests](https://mlsanity.com/paper/2207.04293/) (2022)
- [A Critical Review of Physics-Informed Machine Learning Applications in Subsurface Energy Systems](https://mlsanity.com/paper/2308.04457/) (2023)
- [EarthScape: A Multimodal Dataset for Surficial Geologic Mapping and Earth Surface Analysis](https://mlsanity.com/paper/2503.15625/) (2025)
- [Attention in Diffusion Model: A Survey](https://mlsanity.com/paper/2504.03738/) (2025)
- [AttZoom: Attention Zoom for Better Visual Features](https://mlsanity.com/paper/2508.03625/) (2025)
- [RadarFormer: Lightweight and Accurate Real-Time Radar Object Detection Model](https://mlsanity.com/paper/2304.08447/) (2023)
- [Relational Representation Learning Network for Cross-Spectral Image Patch Matching](https://mlsanity.com/paper/2403.11751/) (2024)
- [mTREE: Multi-Level Text-Guided Representation End-to-End Learning for Whole Slide Image Analysis](https://mlsanity.com/paper/2405.17824/) (2024)
- [Multi-Cohort Framework with Cohort-Aware Attention and Adversarial Mutual-Information Minimization for Whole Slide Image Classification](https://mlsanity.com/paper/2409.11119/) (2024)
- [Revealing Human Attention Patterns from Gameplay Analysis for Reinforcement Learning](https://mlsanity.com/paper/2504.11118/) (2025)
- [Resource-Efficient Glioma Segmentation on Sub-Saharan MRI](https://mlsanity.com/paper/2509.09469/) (2025)
- [Nonparametric Teaching of Attention Learners](https://mlsanity.com/paper/2602.20461/) (2026)
- [Progressive Split Mamba: Effective State Space Modelling for Image Restoration](https://mlsanity.com/paper/2603.09171/) (2026)
- [DyCo-RL: Dynamic Cross-Modal Coordination for Visual Reasoning](https://mlsanity.com/paper/2606.08035/) (2026)
- [Deepening Neural Networks Implicitly and Locally via Recurrent Attention Strategy](https://mlsanity.com/paper/2210.15676/) (2022)
- [3D Detection and Characterisation of ALMA Sources through Deep Learning](https://mlsanity.com/paper/2211.11462/) (2022)
- [Seeing the random forest through the decision trees. Supporting learning health systems from histopathology with machine learning models: Challenges and opportunities](https://mlsanity.com/paper/2312.03812/) (2023)
- [M3BUNet: Mobile Mean Max UNet for Pancreas Segmentation on CT-Scans](https://mlsanity.com/paper/2401.10419/) (2024)
- [Studying the Effects of Self-Attention on SAR Automatic Target Recognition](https://mlsanity.com/paper/2409.00473/) (2024)
- [RemoteTrimmer: Adaptive Structural Pruning for Remote Sensing Image Classification](https://mlsanity.com/paper/2412.12603/) (2024)
- [A recurrent vision transformer shows signatures of primate visual attention](https://mlsanity.com/paper/2502.10955/) (2025)

## Abstract (arXiv)

Inspired by the human cognitive system, attention is a mechanism that imitates the human cognitive awareness about specific information, amplifying critical details to focus more on the essential aspects of data. Deep learning has employed attention to boost performance for many applications. Interestingly, the same attention design can suit processing different data modalities and can easily be incorporated into large networks. Furthermore, multiple complementary attention mechanisms can be incorporated into one network. Hence, attention techniques have become extremely attractive. However, the literature lacks a comprehensive survey on attention techniques to guide researchers in employing attention in their deep models. Note that, besides being demanding in terms of training data and computational resources, transformers only cover a single category in self-attention out of the many categories available. We fill this gap and provide an in-depth survey of 50 attention techniques, categorizing them by their most prominent features. We initiate our discussion by introducing the fundamental concepts behind the success of the attention mechanism. Next, we furnish some essentials such as the strengths and limitations of each attention category, describe their fundamental building blocks, basic formulations with primary usage, and applications specifically for computer vision. We also discuss the challenges and general open questions related to attention mechanisms. Finally, we recommend possible future research directions for deep attention. All the information about visual attention methods in deep learning is provided at hrefhttps://github.com/saeed-anwar/VisualAttentionhttps://github.com/saeed-anwar/VisualAttention