mlsanityData to 7 Oct 2026

Towards Compatible Fine-tuning for Vision-Language Model Updates

arXiv 2412.20895 ↗First posted 30 Dec 2024cs.CV cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

So far, efficient fine-tuning has become a popular strategy for enhancing the capabilities of foundation models on downstream tasks by learning plug-and-play modules. However, existing methods overlook a crucial issue: if the underlying foundation model is updated, are these plug-and-play modules still effective? In this paper, we first conduct a detailed analysis of various fine-tuning methods on the CLIP in terms of their compatibility with model updates. The study reveals that many high-performing fine-tuning methods fail to be compatible with the upgraded models. To address this, we propose a novel approach, Class-conditioned Context Optimization (ContCoOp), which integrates learnable prompts with class embeddings using an attention layer before inputting them into the text encoder. Consequently, the prompts can dynamically adapt to the changes in embedding space (due to model updates), ensuring continued effectiveness. Extensive experiments over 15 datasets show that our ContCoOp achieves the highest compatibility over the baseline methods, and exhibits robust out-of-distribution generalization.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 6 of its 51 references that are in the map. First posted on arXiv 30 Dec 2024.

Builds on

6 of 51 references.

References · 6

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  4. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  5. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  6. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Localized Latent Updates for Fine-Tuning Vision-Language Models
  2. 2025A Closer Look at Conditional Prompt Tuning for Vision-Language Models
  3. 2025Rethinking Fine-Tuning: Unlocking Hidden Capabilities in Vision-Language Models
  4. 2024Connecting the Dots: Collaborative Fine-tuning for Black-Box Vision-Language Models
  5. 2024Can Better Text Semantics in Prompt Tuning Improve VLM Generalization?
  6. 2026Towards Calibrating Prompt Tuning of Vision-Language Models
  7. 2023Distribution-Aware Prompt Tuning for Vision-Language Models
  8. 2026Towards Fine-Grained Robustness: Attention-Guided Test-Time Prompt Tuning for Vision-Language Models