# Vision Transformer using Low-level Chest X-ray Feature Corpus for COVID-19 Diagnosis and Severity Quantification

arXiv: https://arxiv.org/abs/2104.07235 · HTML: https://mlsanity.com/paper/2104.07235/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 69 references that are in the map.
- First posted on arXiv 15 Apr 2021.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [Transformer Interpretability Beyond Attention Visualization](https://mlsanity.com/paper/2012.09838/) (2020)
- [Confounding variables can degrade generalization performance of radiological deep learning models](https://mlsanity.com/paper/1807.00431/) (2018)
- [COVID-Net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest X-Ray Images](https://mlsanity.com/paper/2003.09871/) (2020)
- [Full-Gradient Representation for Neural Network Visualization](https://mlsanity.com/paper/1905.00780/) (2019)
- [Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans](https://mlsanity.com/paper/2008.06388/) (2020)
- [COVID-19 Image Data Collection: Prospective Predictions Are the Future](https://mlsanity.com/paper/2006.11988/) (2020)
- [Covid-19: Automatic detection from X-Ray images utilizing Transfer Learning with Convolutional Neural Networks](https://mlsanity.com/paper/2003.11617/) (2020)
- [Automatic Detection of Coronavirus Disease (COVID-19) Using X-ray Images and Deep Convolutional Neural Networks](https://mlsanity.com/paper/2003.10849/) (2020)
- [BIMCV COVID-19+: a large annotated dataset of RX and CT images from COVID-19 patients](https://mlsanity.com/paper/2006.01174/) (2020)
- [CoroNet: A deep neural network for detection and diagnosis of COVID-19 from chest x-ray images](https://mlsanity.com/paper/2004.04931/) (2020)
- [Deep Learning COVID-19 Features on CXR using Limited Training Data Sets](https://mlsanity.com/paper/2004.05758/) (2020)
- [Deep-COVID: Predicting COVID-19 From Chest X-Ray Images Using Deep Transfer Learning](https://mlsanity.com/paper/2004.09363/) (2020)
- [COVIDX-Net: A Framework of Deep Learning Classifiers to Diagnose COVID-19 in X-Ray Images](https://mlsanity.com/paper/2003.11055/) (2020)
- [Understanding Individual Decisions of CNNs via Contrastive Backpropagation](https://mlsanity.com/paper/1812.02100/) (2018)
- [Weakly Supervised Lesion Localization With Probabilistic-CAM Pooling](https://mlsanity.com/paper/2005.14480/) (2020)

## Built on by (most cited first, 21 of 21)

- [Towards Evaluating Explanations of Vision Transformers for Medical Imaging](https://mlsanity.com/paper/2304.06133/) (2023)
- [Edge-MoE: Memory-Efficient Multi-Task Vision Transformer Architecture with Task-level Sparsity via Mixture-of-Experts](https://mlsanity.com/paper/2305.18691/) (2023)
- [Vision Transformers in Medical Computer Vision -- A Contemplative Retrospection](https://mlsanity.com/paper/2203.15269/) (2022)
- [Federated Split Vision Transformer for COVID-19 CXR Diagnosis using Task-Agnostic Training](https://mlsanity.com/paper/2111.01338/) (2021)
- [A survey on attention mechanisms for medical applications: are we moving towards better algorithms?](https://mlsanity.com/paper/2204.12406/) (2022)
- [Multi-Task Distributed Learning using Vision Transformer with Random Patch Permutation](https://mlsanity.com/paper/2204.03500/) (2022)
- [SSiT: Saliency-guided Self-supervised Image Transformer for Diabetic Retinopathy Grading](https://mlsanity.com/paper/2210.10969/) (2022)
- [Explainable Knowledge Distillation for On-device Chest X-Ray Classification](https://mlsanity.com/paper/2305.06244/) (2023)
- [Radiomics-Guided Global-Local Transformer for Weakly Supervised Pathology Localization in Chest X-Rays](https://mlsanity.com/paper/2207.04394/) (2022)
- [CMC v2: Towards More Accurate COVID-19 Detection with Discriminative Video Priors](https://mlsanity.com/paper/2211.14557/) (2022)
- [Revisiting Computer-Aided Tuberculosis Diagnosis](https://mlsanity.com/paper/2307.02848/) (2023)
- [Efficient and Accurate Pneumonia Detection Using a Novel Multi-Scale Transformer Approach](https://mlsanity.com/paper/2408.04290/) (2024)
- [TranSOP: Transformer-based Multimodal Classification for Stroke Treatment Outcome Prediction](https://mlsanity.com/paper/2301.10829/) (2023)
- [Deep Reinforcement Learning Framework for Thoracic Diseases Classification via Prior Knowledge Guidance](https://mlsanity.com/paper/2306.01232/) (2023)
- [Multi-Feature Vision Transformer via Self-Supervised Representation Learning for Improvement of COVID-19 Diagnosis](https://mlsanity.com/paper/2208.01843/) (2022)
- [Explainable Image Quality Assessment for Medical Imaging](https://mlsanity.com/paper/2303.14479/) (2023)
- [Lung Infection Severity Prediction Using Transformers with Conditional TransMix Augmentation and Cross-Attention](https://mlsanity.com/paper/2510.06887/) (2025)
- [CXR-LT 2026 Challenge: Projection-Aware Multi-Label and Zero-Shot Chest X-Ray Classification](https://mlsanity.com/paper/2604.02185/) (2026)
- [CXR-LT 2026 Challenge: Multi-Center Long-Tailed and Zero Shot Chest X-ray Classification](https://mlsanity.com/paper/2604.15555/) (2026)
- [Self-accumulative Vision Transformer for Bone Age Assessment Using the Sauvegrain Method](https://mlsanity.com/paper/2303.16557/) (2023)
- [Learning from Compressed CT: Feature Attention Style Transfer and Structured Factorized Projections for Resource-Efficient Medical Image Analysis](https://mlsanity.com/paper/2605.00448/) (2026)

## Abstract (arXiv)

Developing a robust algorithm to diagnose and quantify the severity of COVID-19 using Chest X-ray (CXR) requires a large number of well-curated COVID-19 datasets, which is difficult to collect under the global COVID-19 pandemic. On the other hand, CXR data with other findings are abundant. This situation is ideally suited for the Vision Transformer (ViT) architecture, where a lot of unlabeled data can be used through structural modeling by the self-attention mechanism. However, the use of existing ViT is not optimal, since feature embedding through direct patch flattening or ResNet backbone in the standard ViT is not intended for CXR. To address this problem, here we propose a novel Vision Transformer that utilizes low-level CXR feature corpus obtained from a backbone network that extracts common CXR findings. Specifically, the backbone network is first trained with large public datasets to detect common abnormal findings such as consolidation, opacity, edema, etc. Then, the embedded features from the backbone network are used as corpora for a Transformer model for the diagnosis and the severity quantification of COVID-19. We evaluate our model on various external test datasets from totally different institutions to evaluate the generalization capability. The experimental results confirm that our model can achieve the state-of-the-art performance in both diagnosis and severity quantification tasks with superior generalization capability, which are sine qua non of widespread deployment.