mlsanityData to 7 Oct 2026

Low-Rank Few-Shot Adaptation of Vision-Language Models

arXiv 2405.18541 ↗First posted 28 May 2024cs.CVcited by 103

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Recent progress in the few-shot adaptation of Vision-Language Models (VLMs) has further pushed their generalization capabilities, at the expense of just a few labeled samples within the target downstream task. However, this promising, already quite abundant few-shot literature has focused principally on prompt learning and, to a lesser extent, on adapters, overlooking the recent advances in Parameter-Efficient Fine-Tuning (PEFT). Furthermore, existing few-shot learning methods for VLMs often rely on heavy training procedures and/or carefully chosen, task-specific hyper-parameters, which might impede their applicability. In response, we introduce Low-Rank Adaptation (LoRA) in few-shot learning for VLMs, and show its potential on 11 datasets, in comparison to current state-of-the-art prompt- and adapter-based approaches. Surprisingly, our simple CLIP-LoRA method exhibits substantial improvements, while reducing the training times and keeping the same hyper-parameters in all the target tasks, i.e., across all the datasets and numbers of shots. Certainly, our surprising results do not dismiss the potential of prompt-learning and adapter-based research. However, we believe that our strong baseline could be used to evaluate progress in these emergent subjects in few-shot VLMs.

Key facts. Cited by 103 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 52 of its 65 references that are in the map. First posted on arXiv 28 May 2024.

Builds on

52 of 65 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?3 May
  2. 2023A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models20 Dec
  3. 2023Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuning11 Aug
  4. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimization23 Mar
  5. 2022Task Residual for Tuning Vision-Language Models18 Nov
  6. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models15 Sep
  7. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification19 Jul
  8. 2022Prompt-aligned Gradient for Prompt Tuning30 May
  9. 2022Prompt Distribution Learning6 May
  10. 2022Unsupervised Prompt Learning for Vision-Language Models7 Apr
  11. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  12. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
1 more
  1. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  8. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  9. 2013Describing Textures in the Wildcited by 2,110
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  11. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  12. 2022Visual Prompt Tuningcited by 1,674
27 more
  1. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  2. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  3. 2021Grounded Language-Image Pre-trainingcited by 1,147
  4. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  5. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  6. 2021Robust fine-tuning of zero-shot modelscited by 785
  7. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  8. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  9. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  10. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  11. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  12. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  13. 2017Learning multiple visual domains with residual adapterscited by 553
  14. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  15. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  16. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  17. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  18. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 292
  19. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 229
  20. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171
  21. 2022CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attentioncited by 103
  22. 2023One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuningcited by 84
  23. 2023Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matricescited by 72
  24. 2022LASP: Text-to-Text Optimization for Language-Aware Soft Prompting of Vision & Language Modelscited by 58
  25. 2024QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuningcited by 26
  26. 2021NxMTransformer: Semi-Structured Sparsification for Natural Language Understanding via ADMMcited by 19
  27. 2023Hydra: Multi-head Low-rank Adaptation for Parameter Efficient Fine-tuningcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 52; 2026: 39.

103 papers.

Sort:

By area:

  1. 2024Boosting Vision-Language Models with Transductioncited by 36
  2. 2025RandLoRA: Full-rank parameter-efficient fine-tuning of large modelscited by 34
  3. 2025Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Modelscited by 23
  4. 2024FRoundation: Are Foundation Models Ready for Face Recognition?cited by 22
  5. 2025Realistic Test-Time Adaptation of Vision-Language Modelscited by 14
  6. 2024NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Modelscited by 13
  7. 2024Bayesian-guided Label Mapping for Visual Reprogrammingcited by 12
  8. 2025MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystackscited by 11
  9. 2025Towards Higher Effective Rank in Parameter-efficient Fine-tuning using Khatri--Rao Productcited by 10
  10. 2024YOLO-UniOW: Efficient Universal Open-World Object Detectioncited by 9
  11. 2025Few-Shot Adversarial Low-Rank Fine-Tuning of Vision-Language Modelscited by 9
  12. 2025Sharp Generalization Bounds for Foundation Models with Asymmetric Randomized Low-Rank Adapterscited by 9
38 more
  1. 2024FlipSketch: Flipping Static Drawings to Text-Guided Sketch Animationscited by 8
  2. 2025Prompt Disentanglement via Language Guidance and Representation Alignment for Domain Generalizationcited by 8
  3. 2025LoRA-TTT: Low-Rank Test-Time Training for Vision-Language Modelscited by 7
  4. 2025Distributional Vision-Language Alignment by Cauchy-Schwarz Divergencecited by 7
  5. 2025NS-Net: Decoupling CLIP Semantic Information through NULL-Space for Generalizable AI-Generated Image Detectioncited by 7
  6. 2026Reasoning-Driven Multimodal LLM for Domain Generalizationcited by 7
  7. 2025Online Gaussian Test-Time Adaptation of Vision-Language Modelscited by 6
  8. 2025Fine Tuning without Catastrophic Forgetting via Selective Low Rank Adaptationcited by 6
  9. 2025Exploring Cross-Modal Flows for Few-Shot Learningcited by 6
  10. 2025FAA-CLIP: Federated Adversarial Adaptation of CLIPcited by 5
  11. 2025Come Together, But Not Right Now: A Progressive Strategy to Boost Low-Rank Adaptationcited by 5
  12. 2026MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptationcited by 5
  13. 2026Reclaiming Lost Text Layers for Source-Free Cross-Domain Few-Shot Learningcited by 5
  14. 2025CLIP-SLA: Parameter-Efficient CLIP Adaptation for Continuous Sign Language Recognitioncited by 4
  15. 2025MetaTT: A Global Tensor-Train Adapter for Parameter-Efficient Fine-Tuningcited by 4
  16. 2025Rethinking Few Shot CLIP Benchmarks: A Critical Analysis in the Inductive Settingcited by 4
  17. 2025DGS-Net: Distillation-Guided Gradient Surgery for CLIP Fine-Tuning in AI-Generated Image Detectioncited by 4
  18. 2026Path-Decoupled Hyperbolic Flow Matching for Few-Shot Adaptationcited by 4
  19. 2026Mind the Discriminability Trap in Source-Free Cross-domain Few-shot Learningcited by 4
  20. 2026Interpretable Cross-Domain Few-Shot Learning with Rectified Target-Domain Local Alignmentcited by 4
  21. 2024LatteCLIP: Unsupervised CLIP Fine-Tuning via LMM-Synthetic Textscited by 3
  22. 2025pFedMMA: Personalized Federated Fine-Tuning with Multi-Modal Adapter for Vision-Language Modelscited by 3
  23. 2025GLAD: Generalizable Tuning for Vision-Language Modelscited by 3
  24. 2025CLIP-SVD: Efficient and Interpretable Vision-Language Adaptation via Singular Valuescited by 3
  25. 2025Training-free Source Attribution of AI-generated Images via Resynthesiscited by 3
  26. 2026MedCLIPSeg: Probabilistic Vision-Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentationcited by 3
  27. 2026Understanding and Enforcing Weight Disentanglement in Task Arithmeticcited by 3
  28. 2026HyperFM: An Efficient Hyperspectral Foundation Model with Spectral Groupingcited by 3
  29. 2025FATE: A Prompt-Tuning-Based Semi-Supervised Learning Framework for Extremely Limited Labeled Datacited by 2
  30. 2025Visual Language Models show widespread visual deficits on neuropsychological testscited by 2
  31. 2025Bridging Human Oversight and Black-box Driver Assistance: Vision-Language Models for Predictive Alerting in Lane Keeping Assist Systemscited by 2
  32. 2025DeCAF: Decentralized Consensus-And-Factorization for Low-Rank Adaptation of Foundation Modelscited by 2
  33. 2025Enabling Validation for Robust Few-Shot Recognitioncited by 2
  34. 2025VeFA: Vector-Based Feature Space Adaptation for Robust Model Fine-Tuningcited by 2
  35. 2026Cross-Modal Prototype Alignment and Mixing for Training-Free Few-Shot Classificationcited by 2
  36. 2026Addressing Exacerbated Attention Sink for Source-Free Cross-Domain Few-Shot Learningcited by 2
  37. 2026Whole-Slide Image Analysis under Realistic Few-Shot Annotation Protocolscited by 2
  38. 2024Precision at Scale: Domain-Specific Datasets On-Demandcited by 1

50 most cited of 103 All 103 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Black Box Few-Shot Adaptation for Vision-Language models
  2. 2025Rethinking Few-Shot Adaptation of Vision-Language Models in Two Stages
  3. 2025Complementary Subspace Low-Rank Adaptation of Vision-Language Models for Few-Shot Classification
  4. 2026Your Model Already Knows Don't Teach It, Learn to Ask It: Soft Prompting for Few-Shot Adaptation of Vision-Language Models
  5. 2024Test-Time Low Rank Adaptation via Confidence Maximization for Zero-Shot Generalization of Vision-Language Models
  6. 2025Efficient Few-Shot Continual Learning in Vision-Language Models
  7. 2022Localized Latent Updates for Fine-Tuning Vision-Language Models
  8. 2021A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models