# A Multimodal Approach For Endoscopic VCE Image Classification Using BiomedCLIP-PubMedBERT

arXiv: https://arxiv.org/abs/2410.19944 · HTML: https://mlsanity.com/paper/2410.19944/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 8 references that are in the map.
- First posted on arXiv 25 Oct 2024.

## Builds on (most cited first, 4 of 4)

- [WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentation](https://mlsanity.com/paper/2303.14814/) (2023)
- [AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detection](https://mlsanity.com/paper/2310.18961/) (2023)
- [A ChatGPT Aided Explainable Framework for Zero-Shot Medical Image Diagnosis](https://mlsanity.com/paper/2307.01981/) (2023)
- [Capsule Vision 2024 Challenge: Multi-Class Abnormality Classification for Video Capsule Endoscopy](https://mlsanity.com/paper/2408.04940/) (2024)

## Built on by (most cited first, 1 of 1)

- [Optimizing Gastrointestinal Diagnostics: A CNN-Based Model for VCE Image Classification](https://mlsanity.com/paper/2411.01652/) (2024)

## Abstract (arXiv)

This Paper presents an advanced approach for fine-tuning BiomedCLIP PubMedBERT, a multimodal model, to classify abnormalities in Video Capsule Endoscopy (VCE) frames, aiming to enhance diagnostic efficiency in gastrointestinal healthcare. By integrating the PubMedBERT language model with a Vision Transformer (ViT) to process endoscopic images, our method categorizes images into ten specific classes: angioectasia, bleeding, erosion, erythema, foreign body, lymphangiectasia, polyp, ulcer, worms, and normal. Our workflow incorporates image preprocessing and fine-tunes the BiomedCLIP model to generate high-quality embeddings for both visual and textual inputs, aligning them through similarity scoring for classification. Performance metrics, including classification, accuracy, recall, and F1 score, indicate the models strong ability to accurately identify abnormalities in endoscopic frames, showing promise for practical use in clinical diagnostics.