# A survey on attention mechanisms for medical applications: are we moving towards better algorithms?

arXiv: https://arxiv.org/abs/2204.12406 · HTML: https://mlsanity.com/paper/2204.12406/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 139 of its 232 references that are in the map.
- First posted on arXiv 26 Apr 2022.

## Builds on (most cited first, 30 of 139)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Attention U-Net: Learning Where to Look for the Pancreas](https://mlsanity.com/paper/1804.03999/) (2018)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images](https://mlsanity.com/paper/2201.01266/) (2022)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)

## Built on by (most cited first, 15 of 15)

- [Attention Heads of Large Language Models: A Survey](https://mlsanity.com/paper/2409.03752/) (2024)
- [From Screens to Scenes: A Survey of Embodied AI in Healthcare](https://mlsanity.com/paper/2501.07468/) (2025)
- [Attention Mechanisms in Medical Image Segmentation: A Survey](https://mlsanity.com/paper/2305.17937/) (2023)
- [Attention and Self-Attention in Random Forests](https://mlsanity.com/paper/2207.04293/) (2022)
- [Key Patches Are All You Need: A Multiple Instance Learning Framework For Robust Medical Diagnosis](https://mlsanity.com/paper/2405.01654/) (2024)
- [The Past, Current, and Future of Neonatal Intensive Care Units with Artificial Intelligence](https://mlsanity.com/paper/2302.00225/) (2023)
- [A Novel Vision Transformer with Residual in Self-attention for Biomedical Image Classification](https://mlsanity.com/paper/2306.01594/) (2023)
- [Explainable Artificial Intelligence Techniques for Irregular Temporal Classification of Multidrug Resistance Acquisition in Intensive Care Unit Patients](https://mlsanity.com/paper/2407.17165/) (2024)
- [LARF: Two-level Attention-based Random Forests with a Mixture of Contamination Models](https://mlsanity.com/paper/2210.05168/) (2022)
- [A Survey on Causal Representation Learning and Future Work for Medical Image Analysis](https://mlsanity.com/paper/2210.16034/) (2022)
- [A Survey on Cell Nuclei Instance Segmentation and Classification: Leveraging Context and Attention](https://mlsanity.com/paper/2407.18673/) (2024)
- [RotCAtt-TransUNet++: Novel Deep Neural Network for Sophisticated Cardiac Segmentation](https://mlsanity.com/paper/2409.05280/) (2024)
- [Ensemble-Based Survival Models with the Self-Attended Beran Estimator Predictions](https://mlsanity.com/paper/2506.07933/) (2025)
- [UGPL: Uncertainty-Guided Progressive Learning for Evidence-Based Classification in Computed Tomography](https://mlsanity.com/paper/2507.14102/) (2025)
- [MIND: A Noise-Adaptive Denoising Framework for Medical Images Integrating Multi-Scale Transformer](https://mlsanity.com/paper/2508.07817/) (2025)

## Abstract (arXiv)

The increasing popularity of attention mechanisms in deep learning algorithms for computer vision and natural language processing made these models attractive to other research domains. In healthcare, there is a strong need for tools that may improve the routines of the clinicians and the patients. Naturally, the use of attention-based algorithms for medical applications occurred smoothly. However, being healthcare a domain that depends on high-stake decisions, the scientific community must ponder if these high-performing algorithms fit the needs of medical applications. With this motto, this paper extensively reviews the use of attention mechanisms in machine learning (including Transformers) for several medical applications. This work distinguishes itself from its predecessors by proposing a critical analysis of the claims and potentialities of attention mechanisms presented in the literature through an experimental case study on medical image classification with three different use cases. These experiments focus on the integrating process of attention mechanisms into established deep learning architectures, the analysis of their predictive power, and a visual assessment of their saliency maps generated by post-hoc explanation methods. This paper concludes with a critical analysis of the claims and potentialities presented in the literature about attention mechanisms and proposes future research lines in medical applications that may benefit from these frameworks.