mlsanityData to 7 Oct 2026

TransMed: Transformers Advance Multi-modal Medical Image Classification

arXiv 2103.05940 ↗First posted 10 Mar 2021cs.CVcited by 85

Not on the map. Why?

Abstract (arXiv)

Over the past decade, convolutional neural networks (CNN) have shown very competitive performance in medical image analysis tasks, such as disease classification, tumor segmentation, and lesion detection. CNN has great advantages in extracting local features of images. However, due to the locality of convolution operation, it can not deal with long-range relationships well. Recently, transformers have been applied to computer vision and achieved remarkable success in large-scale datasets. Compared with natural images, multi-modal medical images have explicit and important long-range dependencies, and effective multi-modal fusion strategies can greatly improve the performance of deep models. This prompts us to study transformer-based structures and apply them to multi-modal medical images. Existing transformer-based network architectures require large-scale datasets to achieve better performance. However, medical imaging datasets are relatively small, which makes it difficult to apply pure transformers to medical image analysis. Therefore, we propose TransMed for multi-modal medical image classification. TransMed combines the advantages of CNN and transformer to efficiently extract low-level features of images and establish long-range dependencies between modalities. We evaluated our model for the challenging problem of preoperative diagnosis of parotid gland tumors, and the experimental results show the advantages of our proposed method. We argue that the combination of CNN and transformer has tremendous potential in a large number of medical image analysis tasks. To our best knowledge, this is the first work to apply transformers to medical image classification.

Key facts. Cited by 85 papers in the map (data through 7 Oct 2026). Builds on 29 of its 54 references that are in the map. First posted on arXiv 10 Mar 2021.

Builds on

29 of 54 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  8. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  9. 2017Non-local Neural Networkscited by 2,290
  10. 2021TransUNet: Transformers Make Strong Encoders for Medical Image Segmentationcited by 1,552
  11. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  12. 2017Learning Spatio-Temporal Representation with Pseudo-3D Residual Networkscited by 386
17 more
  1. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  2. 20183D MRI brain tumor segmentation using autoencoder regularizationcited by 324
  3. 2016Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain Lesion Segmentationcited by 263
  4. 2021CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentationcited by 227
  5. 2019MultiResUNet : Rethinking the U-Net Architecture for Multimodal Biomedical Image Segmentationcited by 195
  6. 2020TorchIO: A Python library for efficient loading, preprocessing, augmentation and patch-based sampling of medical images in deep learningcited by 176
  7. 2020A review: Deep learning for medical image segmentation using multi-modality fusioncited by 84
  8. 2018HyperDense-Net: A hyper-densely connected CNN for multi-modal image segmentationcited by 66
  9. 2018Brain Tumor Segmentation and Radiomics Survival Prediction: Contribution to the BRATS 2017 Challengecited by 56
  10. 2020MADGAN: unsupervised Medical Anomaly Detection GAN using multiple adjacent brain MRI slice reconstructioncited by 50
  11. 2020Weakly Supervised Deep Learning for COVID-19 Infection Detection and Classification from CT Imagescited by 26
  12. 2020Simultaneous Left Atrium Anatomy and Scar Segmentations via Deep Learning in Multiview Information with Attentioncited by 16
  13. 2017Joint Sequence Learning and Cross-Modality Convolution for 3D Biomedical Segmentationcited by 15
  14. 2019Automatic Prostate Zonal Segmentation Using Fully Convolutional Network with Feature Pyramid Attentioncited by 10
  15. 2018MMFNet: A Multi-modality MRI Fusion Network for Segmentation of Nasopharyngeal Carcinomacited by 8
  16. 2020Knee Injury Detection using MRI with Efficiently-Layered Network (ELNet)cited by 7
  17. 2017Medical Image Segmentation Based on Multi-Modal Convolutional Neural Network: Study on Image Fusion Schemescited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 14; 2023: 21; 2024: 16; 2025: 18; 2026: 10.

85 papers.

Sort:

By area:

  1. 2022Transformers in Medical Imaging: A Surveycited by 245
  2. 2021ResViT: Residual vision transformers for multi-modal medical image synthesiscited by 177
  3. 2022Trustworthy clinical AI solutions: a unified review of uncertainty quantification in deep learning models for medical image analysiscited by 62
  4. 2023Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Reviewcited by 62
  5. 2021Multi-Modal Transformer for Accelerated MR Imagingcited by 42
  6. 2022Transformers in Medical Image Analysis: A Reviewcited by 41
  7. 2022Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectivescited by 32
  8. 2023Artificial General Intelligence for Medical Imaging Analysiscited by 32
  9. 2021GasHis-Transformer: A Multi-scale Visual Transformer Approach for Gastric Histopathological Image Detectioncited by 26
  10. 2021More than Encoder: Introducing Transformer Decoder to Upsamplecited by 26
  11. 2024I2I-Mamba: Multi-modal medical image synthesis via selective state space modelingcited by 26
  12. 2023HEALNet: Multimodal Fusion for Heterogeneous Biomedical Datacited by 25
38 more
  1. 2024A review of deep learning-based information fusion techniques for multimodal medical image classificationcited by 25
  2. 2025Vision Transformers on the Edge: A Comprehensive Survey of Model Compression and Acceleration Strategiescited by 24
  3. 2024Residual-based Language Models are Free Boosters for Biomedical Imagingcited by 23
  4. 2022Vision Transformers in Medical Computer Vision -- A Contemplative Retrospectioncited by 22
  5. 2023A Recent Survey of Vision Transformers for Medical Image Segmentationcited by 22
  6. 2023Large-scale Long-tailed Disease Diagnosis on Radiology Imagescited by 17
  7. 2025Dino U-Net: Exploiting High-Fidelity Dense Features from Foundation Models for Medical Image Segmentationcited by 16
  8. 2022A survey on attention mechanisms for medical applications: are we moving towards better algorithms?cited by 15
  9. 2023Pretrained ViTs Yield Versatile Representations For Medical Imagescited by 13
  10. 2023UniBrain: Universal Brain MRI Diagnosis with Hierarchical Knowledge-enhanced Pre-trainingcited by 13
  11. 2023Is attention all you need in medical image analysis? A reviewcited by 12
  12. 2022Transformers Improve Breast Cancer Diagnosis from Unregistered Multi-View Mammogramscited by 11
  13. 2022Recent Progress in Transformer-based Medical Image Analysiscited by 9
  14. 2022Temporal Context Matters: Enhancing Single Image Prediction with Disease Progression Representationscited by 8
  15. 2022Vision Transformers in Medical Imaging: A Reviewcited by 8
  16. 2024VM-DDPM: Vision Mamba Diffusion for Medical Image Synthesiscited by 8
  17. 2024Structural Attention: Rethinking Transformer for Unpaired Medical Image Synthesiscited by 8
  18. 2025MedKAN: An Advanced Kolmogorov-Arnold Network for Medical Image Classificationcited by 8
  19. 2023FeSViBS: Federated Split Learning of Vision Transformer with Block Samplingcited by 7
  20. 2022Automatic Segmentation of Head and Neck Tumor: How Powerful Transformers Are?cited by 6
  21. 2023Weakly Supervised Intracranial Hemorrhage Segmentation using Head-Wise Gradient-Infused Self-Attention Maps from a Swin Transformer in Categorical Learningcited by 6
  22. 2024Statistical Test for Attention Map in Vision Transformercited by 6
  23. 2021MutualFormer: Multi-Modality Representation Learning via Cross-Diffusion Attentioncited by 5
  24. 2024Efficient and Accurate Pneumonia Detection Using a Novel Multi-Scale Transformer Approachcited by 5
  25. 2025Medical Image Classification with KAN-Integrated Transformers and Dilated Neighborhood Attentioncited by 5
  26. 2023Interpretable Medical Imagery Diagnosis with Self-Attentive Transformers: A Review of Explainable AI for Health Carecited by 4
  27. 2024Probing the Efficacy of Federated Parameter-Efficient Fine-Tuning of Vision Transformers for Medical Image Classificationcited by 4
  28. 2024DiffCLIP: Few-shot Language-driven Multimodal Classifiercited by 4
  29. 2023TranSOP: Transformer-based Multimodal Classification for Stroke Treatment Outcome Predictioncited by 3
  30. 2025Adaptation of Foundation Models for Medical Image Analysis: Strategies, Challenges, and Future Directionscited by 3
  31. 2021AI assisted method for efficiently generating breast ultrasound screening reportscited by 2
  32. 2022A Comparative Study of Gastric Histopathology Sub-size Image Classification: from Linear Regression to Visual Transformercited by 2
  33. 2022Parotid Gland MRI Segmentation Based on Swin-Unet and Multimodal Imagescited by 2
  34. 2023Efficient Scopeformer: Towards Scalable and Rich Feature Extraction for Intracranial Hemorrhage Detectioncited by 2
  35. 2023D-TrAttUnet: Dual-Decoder Transformer-Based Attention Unet Architecture for Binary and Multi-classes Covid-19 Infection Segmentationcited by 2
  36. 2024D-TrAttUnet: Toward Hybrid CNN-Transformer Architecture for Generic and Subtle Segmentation in Medical Imagescited by 2
  37. 2024Understanding differences in applying DETR to natural and medical imagescited by 2
  38. 2025Unleashing Foundation Vision Models: Adaptive Transfer for Diverse Data-Limited Scientific Domainscited by 2

50 most cited of 85 All 85 (JSON)