# Self-Ensembling Vision Transformer (SEViT) for Robust Medical Image Classification

arXiv: https://arxiv.org/abs/2208.02851 · HTML: https://mlsanity.com/paper/2208.02851/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 32 references that are in the map.
- First posted on arXiv 4 Aug 2022.

## Builds on (most cited first, 22 of 22)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [Adversarial Machine Learning at Scale](https://mlsanity.com/paper/1611.01236/) (2016)
- [Ensemble Adversarial Training: Attacks and Defenses](https://mlsanity.com/paper/1705.07204/) (2017)
- [Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks](https://mlsanity.com/paper/1511.04508/) (2015)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)
- [Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Models](https://mlsanity.com/paper/1805.06605/) (2018)
- [Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionality](https://mlsanity.com/paper/1801.02613/) (2018)
- [MagNet: a Two-Pronged Defense against Adversarial Examples](https://mlsanity.com/paper/1705.09064/) (2017)
- [Understanding Robustness of Transformers for Image Classification](https://mlsanity.com/paper/2103.14586/) (2021)
- [Detecting Adversarial Samples from Artifacts](https://mlsanity.com/paper/1703.00410/) (2017)
- [Transformers in Medical Imaging: A Survey](https://mlsanity.com/paper/2201.09873/) (2022)
- [On the Robustness of Vision Transformers to Adversarial Examples](https://mlsanity.com/paper/2104.02610/) (2021)
- [On the Adversarial Robustness of Vision Transformers](https://mlsanity.com/paper/2103.15670/) (2021)
- [On Improving Adversarial Transferability of Vision Transformers](https://mlsanity.com/paper/2106.04169/) (2021)
- [Reliable Tuberculosis Detection using Chest X-ray with Deep Learning, Segmentation and Visualization](https://mlsanity.com/paper/2007.14895/) (2020)
- [Patch-Fool: Are Vision Transformers Always Robust Against Adversarial Perturbations?](https://mlsanity.com/paper/2203.08392/) (2022)
- [E2Net: An Edge Enhanced Network for Accurate Liver and Tumor Segmentation on CT Scans](https://mlsanity.com/paper/2007.09791/) (2020)

## Built on by (most cited first, 16 of 16)

- [Meta-Transformer: A Unified Framework for Multimodal Learning](https://mlsanity.com/paper/2307.10802/) (2023)
- [A Comprehensive Survey of Foundation Models in Medicine](https://mlsanity.com/paper/2406.10729/) (2024)
- [Survey on Adversarial Attack and Defense for Medical Image Analysis: Methods and Challenges](https://mlsanity.com/paper/2303.14133/) (2023)
- [Recent Progress in Transformer-based Medical Image Analysis](https://mlsanity.com/paper/2208.06643/) (2022)
- [Vision Transformers in Medical Imaging: A Review](https://mlsanity.com/paper/2211.10043/) (2022)
- [FeSViBS: Federated Split Learning of Vision Transformer with Block Sampling](https://mlsanity.com/paper/2306.14638/) (2023)
- [On enhancing the robustness of Vision Transformers: Defensive Diffusion](https://mlsanity.com/paper/2305.08031/) (2023)
- [Certification of Deep Learning Models for Medical Image Segmentation](https://mlsanity.com/paper/2310.03664/) (2023)
- [Improving Robustness and Reliability in Medical Image Classification with Latent-Guided Diffusion and Nested-Ensembles](https://mlsanity.com/paper/2310.15952/) (2023)
- [MedFormer: Hierarchical Medical Vision Transformer with Content-Aware Dual Sparse Selection Attention](https://mlsanity.com/paper/2507.02488/) (2025)
- [Recent Advances in Medical Image Classification](https://mlsanity.com/paper/2506.04129/) (2025)
- [S-E Pipeline: A Vision Transformer (ViT) based Resilient Classification Pipeline for Medical Imaging Against Adversarial Attacks](https://mlsanity.com/paper/2407.17587/) (2024)
- [Long-tailed Medical Diagnosis with Relation-aware Representation Learning and Iterative Classifier Calibration](https://mlsanity.com/paper/2502.03238/) (2025)
- [SEDA: Self-Ensembling ViT with Defensive Distillation and Adversarial Training for robust Chest X-rays Classification](https://mlsanity.com/paper/2308.07874/) (2023)
- [FedSIS: Federated Split Learning with Intermediate Representation Sampling for Privacy-preserving Generalized Face Presentation Attack Detection](https://mlsanity.com/paper/2308.10236/) (2023)
- [CheX-DS: Improving Chest X-ray Image Classification with Ensemble Learning Based on DenseNet and Swin Transformer](https://mlsanity.com/paper/2505.11168/) (2025)

## Abstract (arXiv)

Vision Transformers (ViT) are competing to replace Convolutional Neural Networks (CNN) for various computer vision tasks in medical imaging such as classification and segmentation. While the vulnerability of CNNs to adversarial attacks is a well-known problem, recent works have shown that ViTs are also susceptible to such attacks and suffer significant performance degradation under attack. The vulnerability of ViTs to carefully engineered adversarial samples raises serious concerns about their safety in clinical settings. In this paper, we propose a novel self-ensembling method to enhance the robustness of ViT in the presence of adversarial attacks. The proposed Self-Ensembling Vision Transformer (SEViT) leverages the fact that feature representations learned by initial blocks of a ViT are relatively unaffected by adversarial perturbations. Learning multiple classifiers based on these intermediate feature representations and combining these predictions with that of the final ViT classifier can provide robustness against adversarial attacks. Measuring the consistency between the various predictions can also help detect adversarial samples. Experiments on two modalities (chest X-ray and fundoscopy) demonstrate the efficacy of SEViT architecture to defend against various adversarial attacks in the gray-box (attacker has full knowledge of the target model, but not the defense mechanism) setting. Code: https://github.com/faresmalik/SEViT