# COVID-19 Prognosis via Self-Supervised Representation Learning and Multi-Image Prediction

arXiv: https://arxiv.org/abs/2101.04909 · HTML: https://mlsanity.com/paper/2101.04909/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 38 references that are in the map.
- First posted on arXiv 13 Jan 2021.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning](https://mlsanity.com/paper/1711.05225/) (2017)
- [Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning](https://mlsanity.com/paper/1602.03409/) (2016)
- [Deep Learning COVID-19 Features on CXR using Limited Training Data Sets](https://mlsanity.com/paper/2004.05758/) (2020)
- [An interpretable classifier for high-resolution breast cancer screening images utilizing weakly supervised localization](https://mlsanity.com/paper/2002.07613/) (2020)
- [Momentum Contrastive Learning for Few-Shot COVID-19 Diagnosis from Chest CT Images](https://mlsanity.com/paper/2006.13276/) (2020)

## Built on by (most cited first, 30 of 36)

- [Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing](https://mlsanity.com/paper/2301.04558/) (2023)
- [Joint Learning of Localized Representations from Medical Images and Reports](https://mlsanity.com/paper/2112.02889/) (2021)
- [Self-supervised learning methods and applications in medical imaging analysis: A survey](https://mlsanity.com/paper/2109.08685/) (2021)
- [MedAug: Contrastive learning leveraging patient metadata improves representations for chest X-ray interpretation](https://mlsanity.com/paper/2102.10663/) (2021)
- [Transformers in Medical Image Analysis: A Review](https://mlsanity.com/paper/2202.12165/) (2022)
- [Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectives](https://mlsanity.com/paper/2206.01136/) (2022)
- [Contrastive Learning of Single-Cell Phenotypic Representations for Treatment Classification](https://mlsanity.com/paper/2103.16670/) (2021)
- [Anatomy-aware Self-supervised Learning for Anomaly Detection in Chest Radiographs](https://mlsanity.com/paper/2205.04282/) (2022)
- [An Evaluation of Self-Supervised Pre-Training for Skin-Lesion Analysis](https://mlsanity.com/paper/2106.09229/) (2021)
- [A Review of Predictive and Contrastive Self-supervised Learning for Medical Images](https://mlsanity.com/paper/2302.05043/) (2023)
- [SCALP -- Supervised Contrastive Learning for Cardiopulmonary Disease Classification and Localization in Chest X-rays using Patient Metadata](https://mlsanity.com/paper/2110.14787/) (2021)
- [Time-to-Event Pretraining for 3D Medical Imaging](https://mlsanity.com/paper/2411.09361/) (2024)
- [CASS: Cross Architectural Self-Supervision for Medical Image Analysis](https://mlsanity.com/paper/2206.04170/) (2022)
- [SPCXR: Self-supervised Pretraining using Chest X-rays Towards a Domain Specific Foundation Model](https://mlsanity.com/paper/2211.12944/) (2022)
- [MedMAE: A Self-Supervised Backbone for Medical Imaging Tasks](https://mlsanity.com/paper/2407.14784/) (2024)
- [Performance or Trust? Why Not Both. Deep AUC Maximization with Self-Supervised Learning for COVID-19 Chest X-ray Classifications](https://mlsanity.com/paper/2112.08363/) (2021)
- [Improving Classification Model Performance on Chest X-Rays through Lung Segmentation](https://mlsanity.com/paper/2202.10971/) (2022)
- [DiCoM -- Diverse Concept Modeling towards Enhancing Generalizability in Chest X-Ray Studies](https://mlsanity.com/paper/2402.15534/) (2024)
- [TranSOP: Transformer-based Multimodal Classification for Stroke Treatment Outcome Prediction](https://mlsanity.com/paper/2301.10829/) (2023)
- [Self-supervised Model Based on Masked Autoencoders Advance CT Scans Classification](https://mlsanity.com/paper/2210.05073/) (2022)
- [Efficient Representation Learning for Healthcare with Cross-Architectural Self-Supervision](https://mlsanity.com/paper/2308.10064/) (2023)
- [Enhancing Medical Image Segmentation: Optimizing Cross-Entropy Weights and Post-Processing with Autoencoders](https://mlsanity.com/paper/2308.10488/) (2023)
- [Unsupervised Feature Clustering Improves Contrastive Representation Learning for Medical Image Segmentation](https://mlsanity.com/paper/2211.08557/) (2022)
- [A Trustworthy Framework for Medical Image Analysis with Deep Learning](https://mlsanity.com/paper/2212.02764/) (2022)
- [Learning Generalized Medical Image Representations through Image-Graph Contrastive Pretraining](https://mlsanity.com/paper/2405.09594/) (2024)
- [CheXLearner: Text-Guided Fine-Grained Representation Learning for Progression Detection](https://mlsanity.com/paper/2505.06903/) (2025)
- [STA-Risk: A Deep Dive of Spatio-Temporal Asymmetries for Breast Cancer Risk Prediction](https://mlsanity.com/paper/2505.21699/) (2025)
- [Hierarchical Analysis of Visual COVID-19 Features from Chest Radiographs](https://mlsanity.com/paper/2107.06618/) (2021)
- [Dive into Self-Supervised Learning for Medical Image Analysis: Data, Models and Tasks](https://mlsanity.com/paper/2209.12157/) (2022)
- [A Novel Center-based Deep Contrastive Metric Learning Method for the Detection of Polymicrogyria in Pediatric Brain MRI](https://mlsanity.com/paper/2211.12565/) (2022)

## Abstract (arXiv)

The rapid spread of COVID-19 cases in recent months has strained hospital resources, making rapid and accurate triage of patients presenting to emergency departments a necessity. Machine learning techniques using clinical data such as chest X-rays have been used to predict which patients are most at risk of deterioration. We consider the task of predicting two types of patient deterioration based on chest X-rays: adverse event deterioration (i.e., transfer to the intensive care unit, intubation, or mortality) and increased oxygen requirements beyond 6 L per day. Due to the relative scarcity of COVID-19 patient data, existing solutions leverage supervised pretraining on related non-COVID images, but this is limited by the differences between the pretraining data and the target COVID-19 patient data. In this paper, we use self-supervised learning based on the momentum contrast (MoCo) method in the pretraining phase to learn more general image representations to use for downstream tasks. We present three results. The first is deterioration prediction from a single image, where our model achieves an area under receiver operating characteristic curve (AUC) of 0.742 for predicting an adverse event within 96 hours (compared to 0.703 with supervised pretraining) and an AUC of 0.765 for predicting oxygen requirements greater than 6 L a day at 24 hours (compared to 0.749 with supervised pretraining). We then propose a new transformer-based architecture that can process sequences of multiple images for prediction and show that this model can achieve an improved AUC of 0.786 for predicting an adverse event at 96 hours and an AUC of 0.848 for predicting mortalities at 96 hours. A small pilot clinical study suggested that the prediction accuracy of our model is comparable to that of experienced radiologists analyzing the same information.