mlsanityData to 7 Oct 2026

Application Of Vision-Language Models For Assessing Osteoarthritis Disease Severity

arXiv 2401.06331 ↗First posted 12 Jan 2024cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Osteoarthritis (OA) poses a global health challenge, demanding precise diagnostic methods. Current radiographic assessments are time consuming and prone to variability, prompting the need for automated solutions. The existing deep learning models for OA assessment are unimodal single task systems and they don't incorporate relevant text information such as patient demographics, disease history, or physician reports. This study investigates employing Vision Language Processing (VLP) models to predict OA severity using Xray images and corresponding reports. Our method leverages Xray images of the knee and diverse report templates generated from tabular OA scoring values to train a CLIP (Contrastive Language Image PreTraining) style VLP model. Furthermore, we incorporate additional contrasting captions to enforce the model to discriminate between positive and negative reports. Results demonstrate the efficacy of these models in learning text image representations and their contextual relationships, showcase potential advancement in OA assessment, and establish a foundation for specialized vision language models in medical contexts.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 16 of its 29 references that are in the map. First posted on arXiv 12 Jan 2024.

Builds on

16 of 29 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Vision-Language Modelling For Radiological Imaging and Reports In The Low Data Regime30 Mar

Other references · 15

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  4. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  5. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  6. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
  7. 2023Towards Generalist Biomedical AIcited by 286
  8. 2023Learning to Exploit Temporal Structure for Biomedical Vision-Language Processingcited by 210
  9. 2016Grad-CAM: Why did you say that?cited by 159
  10. 2020A Comparison of Pre-trained Vision-and-Language Models for Multimodal Representation Learning across Medical Images and Reportscited by 35
  11. 2021Deep reinforcement learning with automated label extraction from clinical reports accurately classifies 3D MRI brain volumescited by 12
  12. 2023TCEIP: Text Condition Embedded Regression Network for Dental Implant Position Predictioncited by 11
3 more
  1. 2022Self-Supervised-RCNN for Medical Image Segmentation with Limited Data Annotationcited by 7
  2. 2022Weakly Supervised Medical Image Segmentation With Soft Labels and Noise Robust Losscited by 2
  3. 2022Do Language Models Understand Measurements?cited by 2

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Data-Efficient Fine-Tuning of Vision-Language Models for Diagnosis of Alzheimer's Diseasecited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Evaluating the Diagnostic Classification Ability of Multimodal Large Language Models: Insights from the Osteoarthritis Initiative
  2. 2025Risk Estimation of Knee Osteoarthritis Progression via Predictive Multi-task Modelling from Efficient Diffusion Model using X-ray Images
  3. 2026Knee3DVLM: Dual-Sequence Full-Volume Vision-Language Modeling for Comprehensive Knee MRI Assessment
  4. 2025A Reality Check of Vision-Language Pre-training in Radiology: Have We Progressed Using Text?
  5. 2024Self-supervised vision-langage alignment of deep learning representations for bone X-rays analysis
  6. 2023Enhancing Knee Osteoarthritis severity level classification using diffusion augmented images
  7. 2025MedVision: Benchmarking Quantitative Medical Image Analysis
  8. 2024Specialized curricula for training vision-language models in retinal image analysis