# Learning to Adapt Foundation Model DINOv2 for Capsule Endoscopy Diagnosis

arXiv: https://arxiv.org/abs/2406.10508 · HTML: https://mlsanity.com/paper/2406.10508/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 28 references that are in the map.
- First posted on arXiv 15 Jun 2024.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model](https://mlsanity.com/paper/2401.09417/) (2024)
- [Designing Network Design Spaces](https://mlsanity.com/paper/2003.13678/) (2020)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer](https://mlsanity.com/paper/2110.02178/) (2021)
- [Customized Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.13785/) (2023)
- [Surgical-DINO: Adapter Learning of Foundation Models for Depth Estimation in Endoscopic Surgery](https://mlsanity.com/paper/2401.06013/) (2024)
- [EndoDAC: Efficient Adapting Foundation Model for Self-Supervised Depth Estimation from Any Endoscopic Camera](https://mlsanity.com/paper/2405.08672/) (2024)
- [Rethinking Exemplars for Continual Semantic Segmentation in Endoscopy Scenes: Entropy-based Mini-Batch Pseudo-Replay](https://mlsanity.com/paper/2308.14100/) (2023)
- [Meta-learning with implicit gradients in a few-shot setting for medical image segmentation](https://mlsanity.com/paper/2106.03223/) (2021)
- [LLCaps: Learning to Illuminate Low-Light Capsule Endoscopy with Curved Wavelet Attention and Reverse Diffusion](https://mlsanity.com/paper/2307.02452/) (2023)
- [Deep Reinforcement Learning-Based Control for Stomach Coverage Scanning of Wireless Capsule Endoscopy](https://mlsanity.com/paper/2305.10955/) (2023)
- [EndoOOD: Uncertainty-aware Out-of-distribution Detection in Capsule Endoscopy Diagnosis](https://mlsanity.com/paper/2402.11476/) (2024)
- [Semi-supervised Learning for Segmentation of Bleeding Regions in Video Capsule Endoscopy](https://mlsanity.com/paper/2308.02869/) (2023)

## Built on by (most cited first, 8 of 8)

- [FRoundation: Are Foundation Models Ready for Face Recognition?](https://mlsanity.com/paper/2410.23831/) (2024)
- [FoundPAD: Foundation Models Reloaded for Face Presentation Attack Detection](https://mlsanity.com/paper/2501.02892/) (2025)
- [MADation: Face Morphing Attack Detection with Foundation Models](https://mlsanity.com/paper/2501.03800/) (2025)
- [Benchmarking Foundation Models for Mitotic Figure Classification](https://mlsanity.com/paper/2508.04441/) (2025)
- [Parameter Efficient Fine-tuning for Domain-specific Gastrointestinal Disease Recognition](https://mlsanity.com/paper/2604.10451/) (2026)
- [Improving Robustness of Foundation Models in Domain Adaptation with Soup-Adapters](https://mlsanity.com/paper/2507.05807/) (2025)
- [Analogical Reasoning as a Doctor: A Foundation Model for Gastrointestinal Endoscopy Diagnosis](https://mlsanity.com/paper/2604.05649/) (2026)
- [Parameter-Efficient Fine-Tuning of Large Pretrained Models for Instance Segmentation Tasks](https://mlsanity.com/paper/2606.01947/) (2026)

## Abstract (arXiv)

Foundation models have become prominent in computer vision, achieving notable success in various tasks. However, their effectiveness largely depends on pre-training with extensive datasets. Applying foundation models directly to small datasets of capsule endoscopy images from scratch is challenging. Pre-training on broad, general vision datasets is crucial for successfully fine-tuning our model for specific tasks. In this work, we introduce a simplified approach called Adapt foundation models with a low-rank adaptation (LoRA) technique for easier customization. Our method, inspired by the DINOv2 foundation model, applies low-rank adaptation learning to tailor foundation models for capsule endoscopy diagnosis effectively. Unlike traditional fine-tuning methods, our strategy includes LoRA layers designed to absorb specific surgical domain knowledge. During the training process, we keep the main model (the backbone encoder) fixed and focus on optimizing the LoRA layers and the disease classification component. We tested our method on two publicly available datasets for capsule endoscopy disease classification. The results were impressive, with our model achieving 97.75% accuracy on the Kvasir-Capsule dataset and 98.81% on the Kvasirv2 dataset. Our solution demonstrates that foundation models can be adeptly adapted for capsule endoscopy diagnosis, highlighting that mere reliance on straightforward fine-tuning or pre-trained models from general computer vision tasks is inadequate for such specific applications.