# Self-supervised Learning from 100 Million Medical Images

arXiv: https://arxiv.org/abs/2201.01283 · HTML: https://mlsanity.com/paper/2201.01283/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 57 references that are in the map.
- First posted on arXiv 4 Jan 2022.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [Self-labelling via simultaneous clustering and representation learning](https://mlsanity.com/paper/1911.05371/) (2019)
- [PadChest: A large chest x-ray image dataset with multi-label annotated reports](https://mlsanity.com/paper/1901.07441/) (2019)
- [Contrastive learning of global and local features for medical image segmentation with limited annotations](https://mlsanity.com/paper/2006.10511/) (2020)
- [Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks](https://mlsanity.com/paper/1406.6909/) (2014)
- [Learning Representations for Automatic Colorization](https://mlsanity.com/paper/1603.06668/) (2016)
- [Local Aggregation for Unsupervised Learning of Visual Embeddings](https://mlsanity.com/paper/1903.12355/) (2019)
- [Models Genesis: Generic Autodidactic Models for 3D Medical Image Analysis](https://mlsanity.com/paper/1908.06912/) (2019)
- [Learning to See by Moving](https://mlsanity.com/paper/1505.01596/) (2015)
- [Unsupervised Learning by Predicting Noise](https://mlsanity.com/paper/1704.05310/) (2017)
- [ClusterFit: Improving Generalization of Visual Representations](https://mlsanity.com/paper/1912.03330/) (2019)
- [Unsupervised Deep Learning by Neighbourhood Discovery](https://mlsanity.com/paper/1904.11567/) (2019)
- [Learning Image Representations by Completing Damaged Jigsaw Puzzles](https://mlsanity.com/paper/1802.01880/) (2018)
- [CliqueCNN: Deep Unsupervised Exemplar Learning](https://mlsanity.com/paper/1608.08792/) (2016)

## Built on by (most cited first, 30 of 31)

- [On the Challenges and Perspectives of Foundation Models for Medical Image Analysis](https://mlsanity.com/paper/2306.05705/) (2023)
- [Continual Self-supervised Learning: Towards Universal Multi-modal Medical Data Representation Learning](https://mlsanity.com/paper/2311.17597/) (2023)
- [Computational Pathology at Health System Scale -- Self-Supervised Foundation Models from Three Billion Images](https://mlsanity.com/paper/2310.07033/) (2023)
- [Risk of Bias in Chest Radiography Deep Learning Foundation Models](https://mlsanity.com/paper/2209.02965/) (2022)
- [Scaling up self-supervised learning for improved surgical foundation models](https://mlsanity.com/paper/2501.09436/) (2025)
- [FairTune: Optimizing Parameter Efficient Fine Tuning for Fairness in Medical Image Analysis](https://mlsanity.com/paper/2310.05055/) (2023)
- [Prompt Tuning for Parameter-efficient Medical Image Segmentation](https://mlsanity.com/paper/2211.09233/) (2022)
- [Label-Efficient Deep Learning in Medical Image Analysis: Challenges and Future Directions](https://mlsanity.com/paper/2303.12484/) (2023)
- [Disruptive Autoencoders: Leveraging Low-level features for 3D Medical Image Pre-training](https://mlsanity.com/paper/2307.16896/) (2023)
- [Metadata-enhanced contrastive learning from retinal optical coherence tomography images](https://mlsanity.com/paper/2208.02529/) (2022)
- [One-shot Localization and Segmentation of Medical Images with Foundation Models](https://mlsanity.com/paper/2310.18642/) (2023)
- [Zipfian environments for Reinforcement Learning](https://mlsanity.com/paper/2203.08222/) (2022)
- [Counterfactual contrastive learning: robust representations via causal image synthesis](https://mlsanity.com/paper/2403.09605/) (2024)
- [CASS: Cross Architectural Self-Supervision for Medical Image Analysis](https://mlsanity.com/paper/2206.04170/) (2022)
- [Generating and Weighting Semantically Consistent Sample Pairs for Ultrasound Contrastive Learning](https://mlsanity.com/paper/2212.04097/) (2022)
- [Modality-Agnostic Self-Supervised Learning with Meta-Learned Masked Auto-Encoder](https://mlsanity.com/paper/2310.16318/) (2023)
- [MedMerge: Merging Models for Effective Transfer Learning to Medical Imaging Tasks](https://mlsanity.com/paper/2403.11646/) (2024)
- [One Hyper-Initializer for All Network Architectures in Medical Image Analysis](https://mlsanity.com/paper/2206.03661/) (2022)
- [Exploring Self-Supervised Representation Learning For Low-Resource Medical Image Analysis](https://mlsanity.com/paper/2303.02245/) (2023)
- [BenchMD: A Benchmark for Unified Learning on Medical Images and Sensors](https://mlsanity.com/paper/2304.08486/) (2023)
- [Generation of Radiology Findings in Chest X-Ray by Leveraging Collaborative Knowledge](https://mlsanity.com/paper/2306.10448/) (2023)
- [Efficient Representation Learning for Healthcare with Cross-Architectural Self-Supervision](https://mlsanity.com/paper/2308.10064/) (2023)
- [A Study on Self-Supervised Pretraining for Vision Problems in Gastrointestinal Endoscopy](https://mlsanity.com/paper/2401.06278/) (2024)
- [Exploring connections of spectral analysis and transfer learning in medical imaging](https://mlsanity.com/paper/2407.11379/) (2024)
- [Advancing Medical Image Segmentation: Morphology-Driven Learning with Diffusion Transformer](https://mlsanity.com/paper/2408.00347/) (2024)
- [From pre-training to downstream performance: Does domain-specific pre-training make sense?](https://mlsanity.com/paper/2605.08819/) (2026)
- [Quantitative Imaging Principles Improves Medical Image Learning](https://mlsanity.com/paper/2206.06663/) (2022)
- [Cross-Architectural Positive Pairs improve the effectiveness of Self-Supervised Learning](https://mlsanity.com/paper/2301.12025/) (2023)
- [Self-supervised pseudo-colorizing of masked cells](https://mlsanity.com/paper/2302.05968/) (2023)
- [Anatomically aware dual-hop learning for pulmonary embolism detection in CT pulmonary angiograms](https://mlsanity.com/paper/2303.17593/) (2023)

## Abstract (arXiv)

Building accurate and robust artificial intelligence systems for medical image assessment requires not only the research and design of advanced deep learning models but also the creation of large and curated sets of annotated training examples. Constructing such datasets, however, is often very costly -- due to the complex nature of annotation tasks and the high level of expertise required for the interpretation of medical images (e.g., expert radiologists). To counter this limitation, we propose a method for self-supervised learning of rich image features based on contrastive learning and online feature clustering. For this purpose we leverage large training datasets of over 100,000,000 medical images of various modalities, including radiography, computed tomography (CT), magnetic resonance (MR) imaging and ultrasonography. We propose to use these features to guide model training in supervised and hybrid self-supervised/supervised regime on various downstream tasks. We highlight a number of advantages of this strategy on challenging image assessment problems in radiography, CT and MR: 1) Significant increase in accuracy compared to the state-of-the-art (e.g., AUC boost of 3-7% for detection of abnormalities from chest radiography scans and hemorrhage detection on brain CT); 2) Acceleration of model convergence during training by up to 85% compared to using no pretraining (e.g., 83% when training a model for detection of brain metastases in MR scans); 3) Increase in robustness to various image augmentations, such as intensity variations, rotations or scaling reflective of data variation seen in the field.