# Application Of Vision-Language Models For Assessing Osteoarthritis Disease Severity

arXiv: https://arxiv.org/abs/2401.06331 · HTML: https://mlsanity.com/paper/2401.06331/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 16 of its 29 references that are in the map.
- First posted on arXiv 12 Jan 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [VirTex: Learning Visual Representations from Textual Annotations](https://mlsanity.com/paper/2006.06666/) (2020)
- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [Towards Generalist Biomedical AI](https://mlsanity.com/paper/2307.14334/) (2023)
- [Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing](https://mlsanity.com/paper/2301.04558/) (2023)
- [Grad-CAM: Why did you say that?](https://mlsanity.com/paper/1611.07450/) (2016)
- [A Comparison of Pre-trained Vision-and-Language Models for Multimodal Representation Learning across Medical Images and Reports](https://mlsanity.com/paper/2009.01523/) (2020)
- [Vision-Language Modelling For Radiological Imaging and Reports In The Low Data Regime](https://mlsanity.com/paper/2303.17644/) (2023)
- [Deep reinforcement learning with automated label extraction from clinical reports accurately classifies 3D MRI brain volumes](https://mlsanity.com/paper/2106.09812/) (2021)
- [TCEIP: Text Condition Embedded Regression Network for Dental Implant Position Prediction](https://mlsanity.com/paper/2306.14406/) (2023)
- [Self-Supervised-RCNN for Medical Image Segmentation with Limited Data Annotation](https://mlsanity.com/paper/2207.11191/) (2022)
- [Weakly Supervised Medical Image Segmentation With Soft Labels and Noise Robust Loss](https://mlsanity.com/paper/2209.08172/) (2022)
- [Do Language Models Understand Measurements?](https://mlsanity.com/paper/2210.12694/) (2022)

## Built on by (most cited first, 1 of 1)

- [Data-Efficient Fine-Tuning of Vision-Language Models for Diagnosis of Alzheimer's Disease](https://mlsanity.com/paper/2509.07613/) (2025)

## Abstract (arXiv)

Osteoarthritis (OA) poses a global health challenge, demanding precise diagnostic methods. Current radiographic assessments are time consuming and prone to variability, prompting the need for automated solutions. The existing deep learning models for OA assessment are unimodal single task systems and they don't incorporate relevant text information such as patient demographics, disease history, or physician reports. This study investigates employing Vision Language Processing (VLP) models to predict OA severity using Xray images and corresponding reports. Our method leverages Xray images of the knee and diverse report templates generated from tabular OA scoring values to train a CLIP (Contrastive Language Image PreTraining) style VLP model. Furthermore, we incorporate additional contrasting captions to enforce the model to discriminate between positive and negative reports. Results demonstrate the efficacy of these models in learning text image representations and their contextual relationships, showcase potential advancement in OA assessment, and establish a foundation for specialized vision language models in medical contexts.