# Contrastive Learning with Continuous Proxy Meta-Data for 3D MRI Classification

arXiv: https://arxiv.org/abs/2106.08808 · HTML: https://mlsanity.com/paper/2106.08808/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 31 references that are in the map.
- First posted on arXiv 16 Jun 2021.

## Builds on (most cited first, 21 of 21)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [Contrastive Learning with Hard Negative Samples](https://mlsanity.com/paper/2010.04592/) (2020)
- [Debiased Contrastive Learning](https://mlsanity.com/paper/2007.00224/) (2020)
- [Contrastive learning of global and local features for medical image segmentation with limited annotations](https://mlsanity.com/paper/2006.10511/) (2020)
- [3D Self-Supervised Methods for Medical Imaging](https://mlsanity.com/paper/2006.03829/) (2020)
- [Self-supervised Feature Learning for 3D Medical Images by Playing a Rubik's Cube](https://mlsanity.com/paper/1910.02241/) (2019)
- [Revisiting Rubik's Cube: Self-supervised Learning with Volume-wise Transformation for 3D Medical Image Segmentation](https://mlsanity.com/paper/2007.08826/) (2020)
- [CO2: Consistent Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/2010.02217/) (2020)

## Built on by (most cited first, 30 of 49)

- [Rank-N-Contrast: Learning Continuous Representations for Regression](https://mlsanity.com/paper/2210.01189/) (2022)
- [Dissecting Self-Supervised Learning Methods for Surgical Computer Vision](https://mlsanity.com/paper/2207.00449/) (2022)
- [Label-Only Model Inversion Attacks via Knowledge Transfer](https://mlsanity.com/paper/2310.19342/) (2023)
- [Self-Supervised Pre-Training with Contrastive and Masked Autoencoder Methods for Dealing with Small Datasets in Deep Learning for Medical Imaging](https://mlsanity.com/paper/2308.06534/) (2023)
- [Contrastive learning for regression in multi-site brain age prediction](https://mlsanity.com/paper/2211.08326/) (2022)
- [Robust image representations with counterfactual contrastive learning](https://mlsanity.com/paper/2409.10365/) (2024)
- [Anatomical Foundation Models for Brain MRIs](https://mlsanity.com/paper/2408.07079/) (2024)
- [ViT-AE++: Improving Vision Transformer Autoencoder for Self-supervised Medical Image Representations](https://mlsanity.com/paper/2301.07382/) (2023)
- [Model Inversion Robustness: Can Transfer Learning Help?](https://mlsanity.com/paper/2405.05588/) (2024)
- [Integrating Prior Knowledge in Contrastive Learning with Kernel](https://mlsanity.com/paper/2206.01646/) (2022)
- [CLOC: Contrastive Learning for Ordinal Classification with Multi-Margin N-pair Loss](https://mlsanity.com/paper/2504.17813/) (2025)
- [Building a General SimCLR Self-Supervised Foundation Model Across Neurological Diseases to Advance 3D Brain MRI Diagnoses](https://mlsanity.com/paper/2509.10620/) (2025)
- [TractoSCR: A Novel Supervised Contrastive Regression Framework for Prediction of Neurocognitive Measures Using Multi-Site Harmonized Diffusion MRI Tractography](https://mlsanity.com/paper/2210.07411/) (2022)
- [On the Vulnerability of Skip Connections to Model Inversion Attacks](https://mlsanity.com/paper/2409.01696/) (2024)
- [Conditional Alignment and Uniformity for Contrastive Learning with Continuous Proxy Labels](https://mlsanity.com/paper/2111.05643/) (2021)
- [Brain-Aware Replacements for Supervised Contrastive Learning in Detection of Alzheimer's Disease](https://mlsanity.com/paper/2207.04574/) (2022)
- [Separating common from salient patterns with Contrastive Representation Learning](https://mlsanity.com/paper/2402.11928/) (2024)
- [Semi-weakly Supervised Contrastive Representation Learning for Retinal Fundus Images](https://mlsanity.com/paper/2108.02122/) (2021)
- [Contrastive Graph Learning for Population-based fMRI Classification](https://mlsanity.com/paper/2203.14044/) (2022)
- [Learning to diagnose cirrhosis from radiological and histological labels with joint self and weakly-supervised pretraining strategies](https://mlsanity.com/paper/2302.08427/) (2023)
- [Turning Waste into Wealth: Leveraging Low-Quality Samples for Enhancing Continuous Conditional Generative Adversarial Networks](https://mlsanity.com/paper/2308.10273/) (2023)
- [Contrastive Learning with Adaptive Neighborhoods for Brain Age Prediction on 3D Stiffness Maps](https://mlsanity.com/paper/2408.00527/) (2024)
- [PRETI: Patient-Aware Retinal Foundation Model via Metadata-Guided Representation Learning](https://mlsanity.com/paper/2505.12233/) (2025)
- [Edema Estimation From Facial Images Taken Before and After Dialysis via Contrastive Multi-Patient Pre-Training](https://mlsanity.com/paper/2212.07582/) (2022)
- [Decoupled conditional contrastive learning with variable metadata for prostate lesion detection](https://mlsanity.com/paper/2308.09542/) (2023)
- [A Survey of the Impact of Self-Supervised Pretraining for Diagnostic Tasks with Radiological Images](https://mlsanity.com/paper/2309.02555/) (2023)
- [Not Only Grey Matter: OmniBrain for Robust Multimodal Classification of Alzheimer's Disease](https://mlsanity.com/paper/2507.20872/) (2025)
- [Automatic Discovery of Disease Subgroups by Contrasting with Healthy Controls](https://mlsanity.com/paper/2605.21301/) (2026)
- [Optimizing transformations for contrastive learning in a differentiable framework](https://mlsanity.com/paper/2207.13367/) (2022)
- [Adaptive Contrastive Learning with Dynamic Correlation for Multi-Phase Organ Segmentation](https://mlsanity.com/paper/2210.08652/) (2022)

## Abstract (arXiv)

Traditional supervised learning with deep neural networks requires a tremendous amount of labelled data to converge to a good solution. For 3D medical images, it is often impractical to build a large homogeneous annotated dataset for a specific pathology. Self-supervised methods offer a new way to learn a representation of the images in an unsupervised manner with a neural network. In particular, contrastive learning has shown great promises by (almost) matching the performance of fully-supervised CNN on vision tasks. Nonetheless, this method does not take advantage of available meta-data, such as participant's age, viewed as prior knowledge. Here, we propose to leverage continuous proxy metadata, in the contrastive learning framework, by introducing a new loss called y-Aware InfoNCE loss. Specifically, we improve the positive sampling during pre-training by adding more positive examples with similar proxy meta-data with the anchor, assuming they share similar discriminative semantic features.With our method, a 3D CNN model pre-trained on 104 multi-site healthy brain MRI scans can extract relevant features for three classification tasks: schizophrenia, bipolar diagnosis and Alzheimer's detection. When fine-tuned, it also outperforms 3D CNN trained from scratch on these tasks, as well as state-of-the-art self-supervised methods. Our code is made publicly available here.