# MUSCLE: Multi-task Self-supervised Continual Learning to Pre-train Deep Models for X-ray Images of Multiple Body Parts

arXiv: https://arxiv.org/abs/2310.02000 · HTML: https://mlsanity.com/paper/2310.02000/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 27 references that are in the map.
- First posted on arXiv 3 Oct 2023.

## Builds on (most cited first, 16 of 16)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)
- [Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/1902.06162/) (2019)
- [Explicit Inductive Bias for Transfer Learning with Convolutional Networks](https://mlsanity.com/paper/1802.01483/) (2018)
- [Not-so-supervised: a survey of semi-supervised, multi-instance, and transfer learning in medical image analysis](https://mlsanity.com/paper/1804.06353/) (2018)
- [Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization](https://mlsanity.com/paper/2104.05833/) (2021)
- [A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillation](https://mlsanity.com/paper/1810.13243/) (2018)
- [Adversarial Continual Learning for Multi-Domain Hippocampal Segmentation](https://mlsanity.com/paper/2107.08751/) (2021)

## Built on by (most cited first, 13 of 13)

- [LightM-UNet: Mamba Assists in Lightweight UNet for Medical Image Segmentation](https://mlsanity.com/paper/2403.05246/) (2024)
- [Multi-modal Vision Pre-training for Medical Image Analysis](https://mlsanity.com/paper/2410.10604/) (2024)
- [TPO: Aligning Large Language Models with Multi-branch & Multi-step Preference Trees](https://mlsanity.com/paper/2410.12854/) (2024)
- [Continual Learning in Medical Imaging: A Survey and Practical Analysis](https://mlsanity.com/paper/2405.13482/) (2024)
- [Adult Glioma Segmentation in Sub-Saharan Africa using Transfer Learning on Stratified Finetuning Data](https://mlsanity.com/paper/2412.04111/) (2024)
- [Elastic Multi-Gradient Descent for Parallel Continual Learning](https://mlsanity.com/paper/2401.01054/) (2024)
- [Explanations of Classifiers Enhance Medical Image Segmentation via End-to-end Pre-training](https://mlsanity.com/paper/2401.08469/) (2024)
- [CCSI: Continual Class-Specific Impression for Data-free Class Incremental Learning](https://mlsanity.com/paper/2406.05631/) (2024)
- [IRS: Incremental Relationship-guided Segmentation for Digital Pathology](https://mlsanity.com/paper/2505.22855/) (2025)
- [CUPre: Cross-domain Unsupervised Pre-training for Few-Shot Cell Segmentation](https://mlsanity.com/paper/2310.03981/) (2023)
- [Robust and Explainable Framework to Address Data Scarcity in Diagnostic Imaging](https://mlsanity.com/paper/2407.06566/) (2024)
- [Learning with Preserving for Continual Multitask Learning](https://mlsanity.com/paper/2511.11676/) (2025)
- [MedPEFT-CL: Dual-Phase Parameter-Efficient Continual Learning with Medical Semantic Adapter and Bidirectional Memory Consolidation](https://mlsanity.com/paper/2511.17668/) (2025)

## Abstract (arXiv)

While self-supervised learning (SSL) algorithms have been widely used to pre-train deep models, few efforts [11] have been done to improve representation learning of X-ray image analysis with SSL pre-trained models. In this work, we study a novel self-supervised pre-training pipeline, namely Multi-task Self-super-vised Continual Learning (MUSCLE), for multiple medical imaging tasks, such as classification and segmentation, using X-ray images collected from multiple body parts, including heads, lungs, and bones. Specifically, MUSCLE aggregates X-rays collected from multiple body parts for MoCo-based representation learning, and adopts a well-designed continual learning (CL) procedure to further pre-train the backbone subject various X-ray analysis tasks jointly. Certain strategies for image pre-processing, learning schedules, and regularization have been used to solve data heterogeneity, overfitting, and catastrophic forgetting problems for multi-task/dataset learning in MUSCLE.We evaluate MUSCLE using 9 real-world X-ray datasets with various tasks, including pneumonia classification, skeletal abnormality classification, lung segmentation, and tuberculosis (TB) detection. Comparisons against other pre-trained models [7] confirm the proof-of-concept that self-supervised multi-task/dataset continual pre-training could boost the performance of X-ray image analysis.