mlsanityData to 7 Oct 2026

A survey on attention mechanisms for medical applications: are we moving towards better algorithms?

arXiv 2204.12406 ↗First posted 26 Apr 2022cs.CV cs.AI cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

The increasing popularity of attention mechanisms in deep learning algorithms for computer vision and natural language processing made these models attractive to other research domains. In healthcare, there is a strong need for tools that may improve the routines of the clinicians and the patients. Naturally, the use of attention-based algorithms for medical applications occurred smoothly. However, being healthcare a domain that depends on high-stake decisions, the scientific community must ponder if these high-performing algorithms fit the needs of medical applications. With this motto, this paper extensively reviews the use of attention mechanisms in machine learning (including Transformers) for several medical applications. This work distinguishes itself from its predecessors by proposing a critical analysis of the claims and potentialities of attention mechanisms presented in the literature through an experimental case study on medical image classification with three different use cases. These experiments focus on the integrating process of attention mechanisms into established deep learning architectures, the analysis of their predictive power, and a visual assessment of their saliency maps generated by post-hoc explanation methods. This paper concludes with a critical analysis of the claims and potentialities presented in the literature about attention mechanisms and proposes future research lines in medical applications that may benefit from these frameworks.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 139 of its 232 references that are in the map. First posted on arXiv 26 Apr 2022.

Builds on

139 of 232 references.

References · 139

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2016Densely Connected Convolutional Networkscited by 6,691
  8. 2017Squeeze-and-Excitation Networkscited by 4,763
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  10. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  11. 2020Longformer: The Long-Document Transformercited by 3,362
  12. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
38 more
  1. 2018CBAM: Convolutional Block Attention Modulecited by 2,463
  2. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  3. 2017Non-local Neural Networkscited by 2,290
  4. 2020Reformer: The Efficient Transformercited by 1,658
  5. 2021TransUNet: Transformers Make Strong Encoders for Medical Image Segmentationcited by 1,552
  6. 2017Deformable Convolutional Networkscited by 1,481
  7. 20163D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotationcited by 1,471
  8. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  9. 2017Learning Important Features Through Propagating Activation Differencescited by 1,454
  10. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  11. 2018Attention U-Net: Learning Where to Look for the Pancreascited by 1,420
  12. 2021Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentationcited by 1,218
  13. 2018Sanity Checks for Saliency Mapscited by 1,149
  14. 2018Attention-based Deep Multiple Instance Learningcited by 1,077
  15. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  16. 2021Transformers in Vision: A Surveycited by 1,009
  17. 2022Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Imagescited by 823
  18. 2018Self-Attention Generative Adversarial Networkscited by 809
  19. 2014Deep Learning in Neural Networks: An Overviewcited by 765
  20. 2017Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networkscited by 707
  21. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 666
  22. 2014Neural Turing Machinescited by 630
  23. 2020Transformer Interpretability Beyond Attention Visualizationcited by 620
  24. 2013Generating Sequences With Recurrent Neural Networkscited by 549
  25. 2021TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classificationcited by 549
  26. 2020Captum: A unified and generic model interpretability library for PyTorchcited by 542
  27. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  28. 2018A Benchmark for Interpretability Methods in Deep Neural Networkscited by 467
  29. 2017CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learningcited by 459
  30. 2020ResNeSt: Split-Attention Networkscited by 445
  31. 2021Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysiscited by 418
  32. 2019Selective Kernel Networkscited by 401
  33. 2015Attention-Based Models for Speech Recognitioncited by 400
  34. 2019Attention Augmented Convolutional Networkscited by 363
  35. 2020Generating Radiology Reports via Memory-driven Transformercited by 363
  36. 2019Dynamic Convolution: Attention over Convolution Kernelscited by 355
  37. 2014Recurrent Models of Visual Attentioncited by 344
  38. 2020Visual Transformers: Token-based Image Representation and Processing for Computer Visioncited by 333

50 most cited of 139 All 139 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 3; 2024: 5; 2025: 4.

15 papers.

Sort:

  1. 2024Attention Heads of Large Language Models: A Surveycited by 65
  2. 2025From Screens to Scenes: A Survey of Embodied AI in Healthcarecited by 10
  3. 2023Attention Mechanisms in Medical Image Segmentation: A Surveycited by 9
  4. 2022Attention and Self-Attention in Random Forestscited by 4
  5. 2024Key Patches Are All You Need: A Multiple Instance Learning Framework For Robust Medical Diagnosiscited by 3
  6. 2023The Past, Current, and Future of Neonatal Intensive Care Units with Artificial Intelligencecited by 2
  7. 2023A Novel Vision Transformer with Residual in Self-attention for Biomedical Image Classificationcited by 1
  8. 2024Explainable Artificial Intelligence Techniques for Irregular Temporal Classification of Multidrug Resistance Acquisition in Intensive Care Unit Patientscited by 1
  9. 2022LARF: Two-level Attention-based Random Forests with a Mixture of Contamination Modelscited by 0
  10. 2022A Survey on Causal Representation Learning and Future Work for Medical Image Analysiscited by 0
  11. 2024A Survey on Cell Nuclei Instance Segmentation and Classification: Leveraging Context and Attentioncited by 0
  12. 2024RotCAtt-TransUNet++: Novel Deep Neural Network for Sophisticated Cardiac Segmentationcited by 0
3 more
  1. 2025Ensemble-Based Survival Models with the Self-Attended Beran Estimator Predictionscited by 0
  2. 2025UGPL: Uncertainty-Guided Progressive Learning for Evidence-Based Classification in Computed Tomographycited by 0
  3. 2025MIND: A Noise-Adaptive Denoising Framework for Medical Images Integrating Multi-Scale Transformercited by 0