# Vision Transformers in Medical Computer Vision -- A Contemplative Retrospection

arXiv: https://arxiv.org/abs/2203.15269 · HTML: https://mlsanity.com/paper/2203.15269/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 80 of its 239 references that are in the map.
- First posted on arXiv 29 Mar 2022.

## Builds on (most cited first, 30 of 80)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images](https://mlsanity.com/paper/2004.09666/) (2020)
- [A Survey on Visual Transformer](https://mlsanity.com/paper/2012.12556/) (2020)
- [fastMRI: An Open Dataset and Benchmarks for Accelerated MRI](https://mlsanity.com/paper/1811.08839/) (2018)

## Built on by (most cited first, 22 of 22)

- [Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectives](https://mlsanity.com/paper/2206.01136/) (2022)
- [Multimodal Sentiment Analysis: A Survey](https://mlsanity.com/paper/2305.07611/) (2023)
- [Is attention all you need in medical image analysis? A review](https://mlsanity.com/paper/2307.12775/) (2023)
- [Recent Progress in Transformer-based Medical Image Analysis](https://mlsanity.com/paper/2208.06643/) (2022)
- [Which Transformer to Favor: A Comparative Analysis of Efficiency in Vision Transformers](https://mlsanity.com/paper/2308.09372/) (2023)
- [Vision Transformers in Medical Imaging: A Review](https://mlsanity.com/paper/2211.10043/) (2022)
- [TransUKAN:Computing-Efficient Hybrid KAN-Transformer for Enhanced Medical Image Segmentation](https://mlsanity.com/paper/2409.14676/) (2024)
- [DiCoM -- Diverse Concept Modeling towards Enhancing Generalizability in Chest X-Ray Studies](https://mlsanity.com/paper/2402.15534/) (2024)
- [Diffusion Active Learning: Towards Data-Driven Experimental Design in Computed Tomography](https://mlsanity.com/paper/2504.03491/) (2025)
- [An Empirical Analysis for Zero-Shot Multi-Label Classification on COVID-19 CT Scans and Uncurated Reports](https://mlsanity.com/paper/2309.01740/) (2023)
- [Representation Learning of Lab Values via Masked AutoEncoders](https://mlsanity.com/paper/2501.02648/) (2025)
- [A Comprehensive Analysis of Blockchain Applications for Securing Computer Vision Systems](https://mlsanity.com/paper/2307.06659/) (2023)
- [Multi-Granularity Framework for Unsupervised Representation Learning of Time Series](https://mlsanity.com/paper/2312.07248/) (2023)
- [SCFormer: Structured Channel-wise Transformer with Cumulative Historical State for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2505.02655/) (2025)
- [The Eye as a Window to Systemic Health: A Survey of Retinal Imaging from Classical Techniques to Oculomics](https://mlsanity.com/paper/2505.04006/) (2025)
- [SACO: Sequence-Aware Constrained Optimization Framework for Coupon Distribution in E-commerce](https://mlsanity.com/paper/2508.09198/) (2025)
- [Optimizing Multi-Stuttered Speech Classification: Leveraging Whisper's Encoder for Efficient Parameter Reduction in Automated Assessment](https://mlsanity.com/paper/2406.05784/) (2024)
- [ConvShareViT: Enhancing Vision Transformers with Convolutional Attention Mechanisms for Free-Space Optical Accelerators](https://mlsanity.com/paper/2504.11517/) (2025)
- [ViTaL: A Multimodality Dataset and Benchmark for Multi-pathological Ovarian Tumor Recognition](https://mlsanity.com/paper/2507.04383/) (2025)
- [Towards Foundation Models for Cryo-ET Subtomogram Analysis](https://mlsanity.com/paper/2509.24311/) (2025)
- [Token-UNet: A New Case for Transformers Integration in Efficient and Interpretable 3D UNets for Brain Imaging Segmentation](https://mlsanity.com/paper/2602.20008/) (2026)
- [Do emulated quantum circuits change what CNNs look at? Performance and explainability comparison in medical image classification](https://mlsanity.com/paper/2607.21186/) (2026)

## Abstract (arXiv)

Recent escalation in the field of computer vision underpins a huddle of algorithms with the magnificent potential to unravel the information contained within images. These computer vision algorithms are being practised in medical image analysis and are transfiguring the perception and interpretation of Imaging data. Among these algorithms, Vision Transformers are evolved as one of the most contemporary and dominant architectures that are being used in the field of computer vision. These are immensely utilized by a plenty of researchers to perform new as well as former experiments. Here, in this article we investigate the intersection of Vision Transformers and Medical images and proffered an overview of various ViTs based frameworks that are being used by different researchers in order to decipher the obstacles in Medical Computer Vision. We surveyed the application of Vision transformers in different areas of medical computer vision such as image-based disease classification, anatomical structure segmentation, registration, region-based lesion Detection, captioning, report generation, reconstruction using multiple medical imaging modalities that greatly assist in medical diagnosis and hence treatment process. Along with this, we also demystify several imaging modalities used in Medical Computer Vision. Moreover, to get more insight and deeper understanding, self-attention mechanism of transformers is also explained briefly. Conclusively, we also put some light on available data sets, adopted methodology, their performance measures, challenges and their solutions in form of discussion. We hope that this review article will open future directions for researchers in medical computer vision.