mlsanityData to 7 Oct 2026

Localized Latent Updates for Fine-Tuning Vision-Language Models

arXiv 2212.06556 ↗First posted 13 Dec 2022cs.CV cs.CL cs.LGcited by 1

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Although massive pre-trained vision-language models like CLIP show impressive generalization capabilities for many tasks, still it often remains necessary to fine-tune them for improved performance on specific datasets. When doing so, it is desirable that updating the model is fast and that the model does not lose its capabilities on data outside of the dataset, as is often the case with classical fine-tuning approaches. In this work we suggest a lightweight adapter, that only updates the models predictions close to seen datapoints. We demonstrate the effectiveness and speed of this relatively simple approach in the context of few-shot learning, where our results both on classes seen and unseen during training are comparable with or improve on the state of the art.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 43 references that are in the map. First posted on arXiv 13 Dec 2022.

Builds on

29 of 43 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Prompt-aligned Gradient for Prompt Tuning30 May
  2. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  3. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling6 Nov
  4. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  5. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2013Describing Textures in the Wildcited by 2,110
  9. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  10. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  11. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  12. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
12 more
  1. 2019Natural Adversarial Examplescited by 1,401
  2. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  3. 2021Robust fine-tuning of zero-shot modelscited by 785
  4. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  5. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  6. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  7. 2017Zero-Shot Learning -- The Good, the Bad and the Uglycited by 254
  8. 2013Zero-Shot Learning Through Cross-Modal Transfercited by 237
  9. 2014Evaluation of Output Embeddings for Fine-Grained Image Classificationcited by 166
  10. 2015Label-Embedding for Image Classificationcited by 164
  11. 2016An Empirical Study and Analysis of Generalized Zero-Shot Learning for Object Recognition in the Wildcited by 146
  12. 2015Predicting Deep Zero-Shot Convolutional Neural Networks using Textual Descriptionscited by 64

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Prompt Diffusion Robustifies Any-Modality Prompt Learningcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2023Robust Fine-Tuning of Vision-Language Models for Domain Generalization
  2. 2024Low-Rank Few-Shot Adaptation of Vision-Language Models
  3. 2024Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models
  4. 2024Towards Compatible Fine-tuning for Vision-Language Model Updates
  5. 2023A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models
  6. 2025Rethinking Few-Shot Adaptation of Vision-Language Models in Two Stages
  7. 2025Rethinking Fine-Tuning: Unlocking Hidden Capabilities in Vision-Language Models
  8. 2025ProKeR: A Kernel Perspective on Few-Shot Adaptation of Large Vision-Language Models