mlsanityData to 7 Oct 2026

A Systematic Benchmarking Analysis of Transfer Learning for Medical Image Analysis

arXiv 2108.05930 ↗First posted 12 Aug 2021cs.CV eess.IVcited by 31

Not on the map. Why?

Abstract (arXiv)

Transfer learning from supervised ImageNet models has been frequently used in medical image analysis. Yet, no large-scale evaluation has been conducted to benchmark the efficacy of newly-developed pre-training techniques for medical image analysis, leaving several important questions unanswered. As the first step in this direction, we conduct a systematic study on the transferability of models pre-trained on iNat2021, the most recent large-scale fine-grained dataset, and 14 top self-supervised ImageNet models on 7 diverse medical tasks in comparison with the supervised ImageNet model. Furthermore, we present a practical approach to bridge the domain gap between natural and medical images by continually (pre-)training supervised ImageNet models on medical images. Our comprehensive evaluation yields new insights: (1) pre-trained models on fine-grained data yield distinctive local representations that are more suitable for medical segmentation tasks, (2) self-supervised ImageNet models learn holistic features more effectively than supervised ImageNet models, and (3) continual pre-training can bridge the domain gap between natural and medical images. We hope that this large-scale open evaluation of transfer learning can direct the future research of deep learning for medical imaging. As open science, all codes and pre-trained models are available on our GitHub page https://github.com/JLiangLab/BenchmarkTransferLearning.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 33 of its 45 references that are in the map. First posted on arXiv 12 Aug 2021.

Builds on

33 of 45 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  5. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  6. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  7. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  8. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  9. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  10. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  11. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  12. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
21 more
  1. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 969
  2. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  3. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  4. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 636
  5. 2019Self-labelling via simultaneous clustering and representation learningcited by 525
  6. 2020How Well Do Self-Supervised Models Transfer?cited by 213
  7. 2016Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learningcited by 209
  8. 2019Scaling and Benchmarking Self-Supervised Visual Representation Learningcited by 201
  9. 2021Benchmarking Representation Learning for Natural World Image Collectionscited by 167
  10. 2020What makes instance discrimination good for transfer learning?cited by 111
  11. 2020Learning Attentive Pairwise Interaction for Fine-Grained Classificationcited by 84
  12. 2021Self-Supervised Pretraining Improves Self-Supervised Pretrainingcited by 81
  13. 2021A Broad Study on the Transferability of Visual Representations with Contrastive Learningcited by 78
  14. 2021XProtoNet: Diagnosis in Chest Radiography with Global and Local Explanationscited by 72
  15. 2021Transferable Visual Words: Exploiting the Semantics of Anatomical Patterns for Self-supervised Learningcited by 68
  16. 2020The Devil is in the Channels: Mutual-Channel Loss for Fine-Grained Image Classificationcited by 64
  17. 2020Interpreting Chest X-rays via CNNs that Exploit Hierarchical Disease Dependencies and Uncertainty Labelscited by 56
  18. 2021Supervised Transfer Learning at Scale for Medical Imagingcited by 38
  19. 2020Learning Semantics-enriched Representation via Self-discovery, Self-classification, and Self-restorationcited by 35
  20. 2020Can Semantic Labels Assist Self-Supervised Visual Representation Learning?cited by 16
  21. 2021Chest X-Ray Bone Suppression for Improving Classification of Tuberculosis-Consistent Findingscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 13; 2023: 5; 2024: 7; 2025: 2; 2026: 3.

31 papers.

Sort:

By area:

  1. 2022Delving into Masked Autoencoders for Multi-Label Thorax Disease Classificationcited by 65
  2. 2022DiRA: Discriminative, Restorative, and Adversarial Learning for Self-supervised Medical Image Analysiscited by 48
  3. 2022Robust and Efficient Medical Imaging with Self-Supervisioncited by 40
  4. 2024Representing Part-Whole Hierarchies in Foundation Models by Learning Localizability, Composability, and Decomposability from Anatomy via Self-Supervisioncited by 12
  5. 2023Towards Foundation Models Learned from Anatomy in Medical Imaging via Self-Supervisioncited by 11
  6. 2022Probabilistic Integration of Object Level Annotations in Chest X-ray Classificationcited by 10
  7. 2022Anatomy-Aware Contrastive Representation Learning for Fetal Ultrasoundcited by 8
  8. 2022Vision Transformers in Medical Imaging: A Reviewcited by 8
  9. 2022Self-Supervised-RCNN for Medical Image Segmentation with Limited Data Annotationcited by 7
  10. 2023MoMA: Momentum Contrastive Learning with Multi-head Attention-based Knowledge Distillation for Histopathology Image Analysiscited by 6
  11. 2024Scaling Laws for Galaxy Imagescited by 5
  12. 2024Which Backbone to Use: A Resource-efficient Domain Specific Comparison for Computer Visioncited by 5
19 more
  1. 2021Do we still need ImageNet pre-training in remote sensing scene classification?cited by 4
  2. 2024Learning Low-Rank Feature for Thorax Disease Classificationcited by 4
  3. 2023Revisiting Fine-Tuning Strategies for Self-supervised Medical Imaging Analysiscited by 2
  4. 2024Evaluating Self-Supervised Learning in Medical Imaging: A Benchmark for Robustness, Generalizability, and Multi-Domain Impactcited by 2
  5. 2022Robust Alzheimer's Progression Modeling using Cross-Domain Self-Supervised Deep Learningcited by 1
  6. 2022A Knowledge-based Learning Framework for Self-supervised Pre-training Towards Enhanced Recognition of Biomedical Microscopy Imagescited by 1
  7. 2025From Classification to Cross-Modal Understanding: Leveraging Vision-Language Models for Fine-Grained Renal Pathologycited by 1
  8. 2026From pre-training to downstream performance: Does domain-specific pre-training make sense?cited by 1
  9. 2022Resource-efficient domain adaptive pre-training for medical imagescited by 0
  10. 2022RadTex: Learning Efficient Radiograph Representations from Text Reportscited by 0
  11. 2022Dive into Self-Supervised Learning for Medical Image Analysis: Data, Models and Taskscited by 0
  12. 2022Weakly Supervised Learning with Automated Labels from Radiology Reports for Glioma Change Detectioncited by 0
  13. 2023Self-supervised Learning for Segmentation and Quantification of Dopamine Neurons in Parkinson's Diseasecited by 0
  14. 2023Automatic retrieval of corresponding US views in longitudinal examinationscited by 0
  15. 2024Transfer or Self-Supervised? Bridging the Performance Gap in Medical Imagingcited by 0
  16. 2024Leveraging Self-Supervised Learning for Fetal Cardiac Planes Classification using Ultrasound Scan Videoscited by 0
  17. 2025When Do Domain-Specific Foundation Models Justify Their Cost? A Systematic Evaluation Across Retinal Imaging Taskscited by 0
  18. 2026Multi-Dataset Cross-Domain Knowledge Distillation for Unified Medical Image Segmentation, Classification, and Detectioncited by 0
  19. 2026A Unified Framework for the Detection and Classification of Fatty Pancreas in Ultrasound Imagescited by 0