# AI can evolve without labels: self-evolving vision transformer for chest X-ray diagnosis through knowledge distillation

arXiv: https://arxiv.org/abs/2202.06431 · HTML: https://mlsanity.com/paper/2202.06431/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Distillation for medical imaging, one of 12 directions in Knowledge distillation.
- Builds on 23 of its 41 references that are in the map.
- First posted on arXiv 13 Feb 2022.

## Placed in

- Knowledge distillation / [Distillation for medical imaging](https://mlsanity.com/distill/distillation-for-medical-imaging/)

## Builds on (most cited first, 23 of 23)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning](https://mlsanity.com/paper/1711.05225/) (2017)
- [PadChest: A large chest x-ray image dataset with multi-label annotated reports](https://mlsanity.com/paper/1901.07441/) (2019)
- [Billion-scale semi-supervised learning for image classification](https://mlsanity.com/paper/1905.00546/) (2019)
- [Revisiting Self-Training for Neural Sequence Generation](https://mlsanity.com/paper/1909.13788/) (2019)
- [Learning to Self-Train for Semi-Supervised Few-Shot Classification](https://mlsanity.com/paper/1906.00562/) (2019)
- [BIMCV COVID-19+: a large annotated dataset of RX and CT images from COVID-19 patients](https://mlsanity.com/paper/2006.01174/) (2020)
- [Semi-supervised Medical Image Classification with Relation-driven Self-ensembling Model](https://mlsanity.com/paper/2005.07377/) (2020)
- [Weakly Supervised Lesion Localization With Probabilistic-CAM Pooling](https://mlsanity.com/paper/2005.14480/) (2020)
- [Distill on the Go: Online knowledge distillation in self-supervised learning](https://mlsanity.com/paper/2104.09866/) (2021)
- [A Tour of Unsupervised Deep Learning for Medical Image Analysis](https://mlsanity.com/paper/1812.07715/) (2018)
- [BS-Net: learning COVID-19 pneumonia severity on a large Chest X-Ray dataset](https://mlsanity.com/paper/2006.04603/) (2020)
- [Unsupervised Deep Transfer Feature Learning for Medical Image Classification](https://mlsanity.com/paper/1903.06342/) (2019)
- [Recent Deep Semi-supervised Learning Approaches and Related Works](https://mlsanity.com/paper/2106.11528/) (2021)

## Built on by (most cited first, 17 of 17)

- [Exploring scalable medical image encoders beyond text supervision](https://mlsanity.com/paper/2401.10815/) (2024)
- [Medical Slice Transformer: Improved Diagnosis and Explainability on 3D Medical Images with DINOv2](https://mlsanity.com/paper/2411.15802/) (2024)
- [RadAlign: Advancing Radiology Report Generation with Vision-Language Concept Alignment](https://mlsanity.com/paper/2501.07525/) (2025)
- [A Review of Predictive and Contrastive Self-supervised Learning for Medical Images](https://mlsanity.com/paper/2302.05043/) (2023)
- [Revisiting Computer-Aided Tuberculosis Diagnosis](https://mlsanity.com/paper/2307.02848/) (2023)
- [Medical Diagnosis with Large Scale Multimodal Transformers: Leveraging Diverse Data for More Accurate Diagnosis](https://mlsanity.com/paper/2212.09162/) (2022)
- [Large-scale single-photon imaging](https://mlsanity.com/paper/2212.13654/) (2022)
- [Generate to Ground: Multimodal Text Conditioning Boosts Phrase Grounding in Medical Vision-Language Models](https://mlsanity.com/paper/2507.12236/) (2025)
- [Zero-Shot Pediatric Tuberculosis Detection in Chest X-Rays using Self-Supervised Learning](https://mlsanity.com/paper/2402.14741/) (2024)
- [Low-Resolution Chest X-ray Classification via Knowledge Distillation and Multi-task Learning](https://mlsanity.com/paper/2405.13370/) (2024)
- [Empowering Tuberculosis Screening with Explainable Self-Supervised Deep Neural Networks](https://mlsanity.com/paper/2406.13750/) (2024)
- [A Knowledge Distillation-Based Approach to Enhance Transparency of Classifier Models](https://mlsanity.com/paper/2502.15959/) (2025)
- [CXR-CML: Improved zero-shot classification of long-tailed multi-label diseases in Chest X-Rays](https://mlsanity.com/paper/2507.19398/) (2025)
- [MVC: A Multi-Task Vision Transformer Network for COVID-19 Diagnosis from Chest X-ray Images](https://mlsanity.com/paper/2310.00418/) (2023)
- [DiSSECT: Structuring Transfer-Ready Medical Image Representations through Discrete Self-Supervision](https://mlsanity.com/paper/2509.18765/) (2025)
- [An Explainable Hybrid AI Framework for Enhanced Tuberculosis and Symptom Detection](https://mlsanity.com/paper/2510.18819/) (2025)
- [Weakly Supervised Tuberculosis Localization in Chest X-rays through Knowledge Distillation](https://mlsanity.com/paper/2512.11057/) (2025)

## Abstract (arXiv)

Although deep learning-based computer-aided diagnosis systems have recently achieved expert-level performance, developing a robust deep learning model requires large, high-quality data with manual annotation, which is expensive to obtain. This situation poses the problem that the chest x-rays collected annually in hospitals cannot be used due to the lack of manual labeling by experts, especially in deprived areas. To address this, here we present a novel deep learning framework that uses knowledge distillation through self-supervised learning and self-training, which shows that the performance of the original model trained with a small number of labels can be gradually improved with more unlabeled data. Experimental results show that the proposed framework maintains impressive robustness against a real-world environment and has general applicability to several diagnostic tasks such as tuberculosis, pneumothorax, and COVID-19. Notably, we demonstrated that our model performs even better than those trained with the same amount of labeled data. The proposed framework has a great potential for medical imaging, where plenty of data is accumulated every year, but ground truth annotations are expensive to obtain.