LEAML: Label-Efficient Adaptation to Out-of-Distribution Visual Tasks for Multimodal Large Language Models
On the map: Multimodal LLMs / vision-language models → Medical vision-language models
Abstract (arXiv)
Multimodal Large Language Models (MLLMs) have achieved strong performance on general visual benchmarks but struggle with out-of-distribution (OOD) tasks in specialized domains such as medical imaging, where labeled data is limited and expensive. We introduce LEAML, a label-efficient adaptation framework that leverages both scarce labeled VQA samples and abundant unlabeled images. Our approach generates domain-relevant pseudo question-answer pairs for unlabeled data using a QA generator regularized by caption distillation. Importantly, we selectively update only those neurons most relevant to question-answering, enabling the QA Generator to efficiently acquire domain-specific knowledge during distillation. Experiments on gastrointestinal endoscopy and sports VQA demonstrate that LEAML consistently outperforms standard fine-tuning under minimal supervision, highlighting the effectiveness of our proposed LEAML framework.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 19 of its 28 references that are in the map. First posted on arXiv 3 Oct 2025.
Builds on
19 of 28 references.
References · 19
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2025Qwen2.5-VL Technical Reportcited by 5,587
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
- 2023Qwen Technical Reportcited by 3,344
- 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
- 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
- 2021Knowledge Neurons in Pretrained Transformerscited by 601
- 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
- 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 520
- 2023CLIP-Driven Universal Model for Organ Segmentation and Tumor Detectioncited by 224
- 2022Delving into Out-of-Distribution Detection with Vision-Language Representationscited by 216
7 more
- 2024NVILA: Efficient Frontier Visual Language Modelscited by 209
- 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
- 2023CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say Nocited by 130
- 2024Kvasir-VQA: A Text-Image Pair GI Tract Datasetcited by 23
- 2024VideoICL: Confidence-based Iterative In-context Learning for Out-of-Distribution Video Understandingcited by 17
- 2023CLIP-Lung: Textual Knowledge-Guided Lung Nodule Malignancy Predictioncited by 16
- 2025Locate-then-Merge: Neuron-Level Parameter Fusion for Mitigating Catastrophic Forgetting in Multimodal LLMscited by 7
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024PeFoMed: Parameter Efficient Fine-tuning of Multimodal Large Language Models for Medical Imaging
- 2024LLaVA-Ultra: Large Chinese Language and Vision Assistant for Ultrasound
- 2024Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering
- 2024On Domain-Adaptive Post-Training for Multimodal Large Language Models
- 2025Synthesizing High-Quality Visual Question Answering from Medical Documents with Generator-Verifier LMMs
- 2024Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicine
- 2024MISS: A Generative Pretraining and Finetuning Approach for Med-VQA
- 2026Instruction-Free Tuning of Large Vision Language Models for Medical Instruction Following