Self-Supervised Learning for Knee Osteoarthritis: Diagnostic Limitations and Prognostic Value of Hospital Data
Not on the map. Why?
Abstract (arXiv)
This study assesses whether self-supervised learning (SSL) improves knee osteoarthritis (OA) modeling for diagnosis and prognosis relative to ImageNet-pretrained initialization. We compared (i) image-only SSL pretrained on knee radiographs from the OAI, MOST, and NYU cohorts, and (ii) multimodal image-text SSL pretrained on hospital knee radiographs paired with radiologist impressions. For diagnostic Kellgren-Lawrence (KL) grade prediction, SSL yielded mixed results. While image-only SSL improved accuracy during linear probing (frozen encoder), it did not outperform ImageNet pretraining during full fine-tuning. Similarly, multimodal SSL failed to improve grading performance. A likely explanation is mismatch between the hospital pretraining corpus and the downstream diagnostic task: the hospital image-text dataset was restricted to knees from patients with clinically identified OA in routine care, rather than a cohort spanning the full spectrum from normal to severe disease needed for balanced KL grading. In addition, radiology impressions do not explicitly encode KL grade, limiting supervision for learning KL-specific decision boundaries. In contrast, this same multimodal initialization significantly improved prognostic modeling. It outperformed ImageNet baselines in predicting 4-year structural incidence and progression, including on external validation (MOST AUROC: 0.701 vs. 0.599 at 10% labeled data). Overall, these results suggest that our hospital image-text data may be less effective for diagnostic grading when the pretraining cohort is limited to OA knees, but can provide a strong signal for prognostic modeling when the downstream task is better aligned with the pretraining data distribution.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 20 of its 38 references that are in the map. First posted on arXiv 26 Mar 2026.
Builds on
20 of 38 references.
References · 20
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
- 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
- 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
- 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
- 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
- 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
- 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
8 more
- 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 969
- 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
- 2021Big Self-Supervised Models Advance Medical Image Classificationcited by 320
- 2017Automatic Knee Osteoarthritis Diagnosis from Plain Radiographs: A Deep Learning-Based Approachcited by 42
- 2020A new paradigm for accelerating clinical data science at Stanford Medicinecited by 16
- 2020Deep learning predicts total knee replacement from magnetic resonance imagescited by 10
- 2024CNN-JEPA: Self-Supervised Pretraining Convolutional Neural Networks Using Joint Embedding Predictive Architecturecited by 5
- 2024A Progressive Risk Formulation for Enhanced Deep Learning based Total Knee Replacement Prediction in Knee Osteoarthritiscited by 1
Built on by
Not cited yet.