mlsanityData to 14 Oct 2026

Multi-Modal Adapter for Vision-Language Models

arXiv 2409.02958 ↗First posted 3 Sep 2024cs.CV cs.AI cs.LGcited by 8

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Large pre-trained vision-language models, such as CLIP, have demonstrated state-of-the-art performance across a wide range of image classification tasks, without requiring retraining. Few-shot CLIP is competitive with existing specialized architectures that were trained on the downstream tasks. Recent research demonstrates that the performance of CLIP can be further improved using lightweight adaptation approaches. However, previous methods adapt different modalities of the CLIP model individually, ignoring the interactions and relationships between visual and textual representations. In this work, we propose Multi-Modal Adapter, an approach for Multi-Modal adaptation of CLIP. Specifically, we add a trainable Multi-Head Attention layer that combines text and image features to produce an additive adaptation of both. Multi-Modal Adapter demonstrates improved generalizability, based on its performance on unseen classes compared to existing adaptation methods. We perform additional ablations and investigations to validate and interpret the proposed approach.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 37 references that are in the map. First posted on arXiv 3 Sep 2024.

Builds on

26 of 37 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Improving Zero-Shot Generalization for CLIP with Synthesized Prompts14 Jul
  2. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  3. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling6 Nov
  4. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  5. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 21

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  12. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
9 more
  1. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  2. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  3. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  4. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  5. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  6. 2017Learning multiple visual domains with residual adapterscited by 557
  7. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
  8. 2024Any-Shift Prompting for Generalization over Distributionscited by 24
  9. 2024Cross-Modal Adapter: Parameter-Efficient Transfer Learning Approach for Vision-Language Modelscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 5.

8 papers.

  1. 2025DPC: Dual-Prompt Collaboration for Tuning Vision-Language Modelscited by 18
  2. 2025CXR-CML: Improved zero-shot classification of long-tailed multi-label diseases in Chest X-Rayscited by 1
  3. 2025What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Mergingcited by 1
  4. 2026DeAR: Fine-Grained VLM Adaptation by Decomposing Attention Head Rolescited by 1
  5. 2026FVG-PT: Adaptive Foreground View-Guided Prompt Tuning for Vision-Language Modelscited by 1
  6. 2026Gram-Anchored Prompt Learning for Vision-Language Models via Second-Order Statisticscited by 0
  7. 2026Cross-Stage Coherence in Hierarchical Driving VQA: Explicit Baselines and Learned Gated Context Projectorscited by 0
  8. 2026MQAdapter: Multi-Modal Quantum Adapter for Coarse-to-Fine VLM Fine-tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025CLIP-SVD: Efficient and Interpretable Vision-Language Adaptation via Singular Values
  2. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification
  3. 2025RMAdapter: Reconstruction-based Multi-Modal Adapter for Vision-Language Models
  4. 2022SVL-Adapter: Self-Supervised Adapter for Vision-Language Pretrained Models
  5. 2023Meta-Adapter: An Online Few-shot Learner for Vision-Language Model
  6. 2024CLIPArTT: Adaptation of CLIP to New Domains at Test Time
  7. 2024Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models
  8. 2025Post-pre-training for Modality Alignment in Vision-Language Foundation Models