mlsanityData to 14 Oct 2026

Early Intervention for VFM-based Multimodal Medical Image Classification

arXiv 2603.17514 ↗First posted 18 Mar 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Current methods for multimodal medical image classification (M3IC) face two major challenges. First, the prevailing "fusion after unimodal image embedding" paradigm cannot fully exploit the complementary and correlated information in the multimodal data. Second, the scarcity of labeled multimodal medical images, coupled with their substantial domain shift from natural images, impedes the use of cutting-edge Vision Foundation Models (VFMs) for medical image embedding. To jointly address the challenges, we propose a novel Early Intervention (EI) framework. Treating one modality as target and the rest as reference, EI harnesses high-level semantic tokens from the reference as intervention tokens to steer the target modality's embedding process at an early stage. For parameter-efficient VFM adaptation, we introduce Mixture of varied-rank LoRAs (MoR), which employs a small set of LoRA experts with distinct ranks and a bypass-allowed router. Extensive experiments on three public datasets, covering retinal disease recognition, skin lesion recognition, and knee anomaly classification, verify the effectiveness of the proposed method against a number of competitive baselines.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 50 references that are in the map. First posted on arXiv 18 Mar 2026.

Builds on

31 of 50 references.

References · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  4. 2022Visual Prompt Tuningcited by 1,681
  5. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  6. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  7. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  8. 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 141
  9. 2023Sparse Low-rank Adaptation of Pre-trained Language Modelscited by 96
  10. 2023DFormer: Rethinking RGBD Representation Learning for Semantic Segmentationcited by 60
  11. 2022Dynamic Multimodal Fusioncited by 55
  12. 2024The Impact of Initialization on LoRA Finetuning Dynamicscited by 54
19 more
  1. 2024A Multimodal Vision Foundation Model for Clinical Dermatologycited by 50
  2. 2025RandLoRA: Full-rank parameter-efficient fine-tuning of large modelscited by 35
  3. 2025FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Expertscited by 35
  4. 2025The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformercited by 30
  5. 2024Time-, Memory- and Parameter-Efficient Visual Adaptationcited by 28
  6. 2024Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approachcited by 18
  7. 2023Large-scale Long-tailed Disease Diagnosis on Radiology Imagescited by 17
  8. 2025Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamicscited by 16
  9. 2020Learning Two-Stream CNN for Multi-Modal Age-related Macular Degeneration Categorizationcited by 15
  10. 2021Multi-Modal Multi-Instance Learning for Retinal Disease Recognitioncited by 14
  11. 2022SFusion: Self-attention based N-to-One Multimodal Fusion Blockcited by 14
  12. 2025Rethinking Multimodal Learning from the Perspective of Mitigating Classification Ability Disproportioncited by 13
  13. 2025STiL: Semi-supervised Tabular-Image Learning for Comprehensive Task-Relevant Information Exploration in Multimodal Classificationcited by 12
  14. 2023DVPT: Dynamic Visual Prompt Tuning of Large Pre-trained Models for Medical Image Analysiscited by 11
  15. 2025UORA: Uniform Orthogonal Reinitialization Adaptation in Parameter-Efficient Fine-Tuning of Large Modelscited by 11
  16. 2024LoLDU: Low-Rank Adaptation via Lower-Diag-Upper Decomposition for Parameter-Efficient Fine-Tuningcited by 6
  17. 2024Prompt Your Brain: Scaffold Prompt Tuning for Efficient Adaptation of fMRI Pre-trained Modelcited by 3
  18. 2024Boosting Memory Efficiency in Transfer Learning for High-Resolution Medical Image Classificationcited by 2
  19. 2025LKA: Large Kernel Adapter for Enhanced Medical Image Classificationcited by 1

Built on by

Not cited yet.