# MedAug: Contrastive learning leveraging patient metadata improves representations for chest X-ray interpretation

arXiv: https://arxiv.org/abs/2102.10663 · HTML: https://mlsanity.com/paper/2102.10663/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 18 references that are in the map.
- First posted on arXiv 21 Feb 2021.

## Builds on (most cited first, 16 of 16)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [Hard Negative Mixing for Contrastive Learning](https://mlsanity.com/paper/2010.01028/) (2020)
- [Big Self-Supervised Models Advance Medical Image Classification](https://mlsanity.com/paper/2101.05224/) (2021)
- [Contrastive learning of global and local features for medical image segmentation with limited annotations](https://mlsanity.com/paper/2006.10511/) (2020)
- [CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patients](https://mlsanity.com/paper/2005.13249/) (2020)
- [Viewmaker Networks: Learning Views for Unsupervised Representation Learning](https://mlsanity.com/paper/2010.07432/) (2020)
- [COVID-19 Prognosis via Self-Supervised Representation Learning and Multi-Image Prediction](https://mlsanity.com/paper/2101.04909/) (2021)
- [Deep Learning for the Digital Pathologic Diagnosis of Cholangiocarcinoma and Hepatocellular Carcinoma: Evaluating the Impact of a Web-based Diagnostic Assistant](https://mlsanity.com/paper/1911.07372/) (2019)

## Built on by (most cited first, 30 of 52)

- [BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs](https://mlsanity.com/paper/2303.00915/) (2023)
- [Transformers in Medical Imaging: A Survey](https://mlsanity.com/paper/2201.09873/) (2022)
- [Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing](https://mlsanity.com/paper/2301.04558/) (2023)
- [Recent advances and clinical applications of deep learning in medical image analysis](https://mlsanity.com/paper/2105.13381/) (2021)
- [CXR-CLIP: Toward Large Scale Chest X-ray Language-Image Pre-training](https://mlsanity.com/paper/2310.13292/) (2023)
- [Self-supervised learning methods and applications in medical imaging analysis: A survey](https://mlsanity.com/paper/2109.08685/) (2021)
- [PCRLv2: A Unified Visual Information Preservation Framework for Self-supervised Pre-training in Medical Image Analysis](https://mlsanity.com/paper/2301.00772/) (2023)
- [3KG: Contrastive Learning of 12-Lead Electrocardiograms using Physiologically-Inspired Augmentations](https://mlsanity.com/paper/2106.04452/) (2021)
- [Intermediate Layers Matter in Momentum Contrastive Self Supervised Learning](https://mlsanity.com/paper/2110.14805/) (2021)
- [Sequential Multi-Dimensional Self-Supervised Learning for Clinical Time Series](https://mlsanity.com/paper/2307.10923/) (2023)
- [Pretraining Respiratory Sound Representations using Metadata and Contrastive Learning](https://mlsanity.com/paper/2210.16192/) (2022)
- [Significantly improving zero-shot X-ray pathology classification via fine-tuning pre-trained image-text encoders](https://mlsanity.com/paper/2212.07050/) (2022)
- [Metadata-enhanced contrastive learning from retinal optical coherence tomography images](https://mlsanity.com/paper/2208.02529/) (2022)
- [Robust image representations with counterfactual contrastive learning](https://mlsanity.com/paper/2409.10365/) (2024)
- [Imbalance-Aware Self-Supervised Learning for 3D Radiomic Representations](https://mlsanity.com/paper/2103.04167/) (2021)
- [Contrastive Learning of Single-Cell Phenotypic Representations for Treatment Classification](https://mlsanity.com/paper/2103.16670/) (2021)
- [Anatomy-aware Self-supervised Learning for Anomaly Detection in Chest Radiographs](https://mlsanity.com/paper/2205.04282/) (2022)
- [An Evaluation of Self-Supervised Pre-Training for Skin-Lesion Analysis](https://mlsanity.com/paper/2106.09229/) (2021)
- [SCALP -- Supervised Contrastive Learning for Cardiopulmonary Disease Classification and Localization in Chest X-rays using Patient Metadata](https://mlsanity.com/paper/2110.14787/) (2021)
- [Archangel: A Hybrid UAV-based Human Detection Benchmark with Position and Pose Metadata](https://mlsanity.com/paper/2209.00128/) (2022)
- [Clinically Labeled Contrastive Learning for OCT Biomarker Classification](https://mlsanity.com/paper/2305.15154/) (2023)
- [Anatomy-Aware Contrastive Representation Learning for Fetal Ultrasound](https://mlsanity.com/paper/2208.10642/) (2022)
- [Region-based Contrastive Pretraining for Medical Image Retrieval with Anatomic Query](https://mlsanity.com/paper/2305.05598/) (2023)
- [Data efficient deep learning for medical image analysis: A survey](https://mlsanity.com/paper/2310.06557/) (2023)
- [Chest X-ray Foundation Model with Global and Local Representations Integration](https://mlsanity.com/paper/2502.05142/) (2025)
- [Generating and Weighting Semantically Consistent Sample Pairs for Ultrasound Contrastive Learning](https://mlsanity.com/paper/2212.04097/) (2022)
- [BSDA: Bayesian Random Semantic Data Augmentation for Medical Image Classification](https://mlsanity.com/paper/2403.06138/) (2024)
- [MedSelect: Selective Labeling for Medical Image Classification Combining Meta-Learning with Deep Reinforcement Learning](https://mlsanity.com/paper/2103.14339/) (2021)
- [Can we Adopt Self-supervised Pretraining for Chest X-Rays?](https://mlsanity.com/paper/2211.12931/) (2022)
- [GCL: Gradient-Guided Contrastive Learning for Medical Image Segmentation with Multi-Perspective Meta Labels](https://mlsanity.com/paper/2309.08888/) (2023)

## Abstract (arXiv)

Self-supervised contrastive learning between pairs of multiple views of the same image has been shown to successfully leverage unlabeled data to produce meaningful visual representations for both natural and medical images. However, there has been limited work on determining how to select pairs for medical images, where availability of patient metadata can be leveraged to improve representations. In this work, we develop a method to select positive pairs coming from views of possibly different images through the use of patient metadata. We compare strategies for selecting positive pairs for chest X-ray interpretation including requiring them to be from the same patient, imaging study or laterality. We evaluate downstream task performance by fine-tuning the linear layer on 1% of the labeled dataset for pleural effusion classification. Our best performing positive pair selection strategy, which involves using images from the same patient from the same study across all lateralities, achieves a performance increase of 14.4% in mean AUC from the ImageNet pretrained baseline. Our controlled experiments show that the keys to improving downstream performance on disease classification are (1) using patient metadata to appropriately create positive pairs from different images with the same underlying pathologies, and (2) maximizing the number of different images used in query pairing. In addition, we explore leveraging patient metadata to select hard negative pairs for contrastive learning, but do not find improvement over baselines that do not use metadata. Our method is broadly applicable to medical image interpretation and allows flexibility for incorporating medical insights in choosing pairs for contrastive learning.