mlsanityData to 7 Oct 2026

Making the Most of What You Have: Adapting Pre-trained Visual Language Models in the Low-data Regime

arXiv 2305.02297 ↗First posted 3 May 2023cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Large-scale visual language models are widely used as pre-trained models and then adapted for various downstream tasks. While humans are known to efficiently learn new tasks from a few examples, deep learning models struggle with adaptation from few examples. In this work, we look into task adaptation in the low-data regime, and provide a thorough study of the existing adaptation methods for generative Visual Language Models. And we show important benefits of self-labelling, i.e. using the model's own predictions to self-improve when having access to a larger number of unlabelled images of the same distribution. Our study demonstrates significant gains using our proposed task adaptation pipeline across a wide range of visual language tasks such as visual classification (ImageNet), visual captioning (COCO), detailed visual captioning (Localised Narratives) and visual question answering (VQAv2).

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 66 of its 73 references that are in the map. First posted on arXiv 3 May 2023.

Builds on

66 of 73 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 65

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2016Layer Normalizationcited by 5,579
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
38 more
  1. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  2. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  3. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  4. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  5. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  6. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  7. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  8. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  9. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  10. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  11. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  12. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  13. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  14. 2021Robust fine-tuning of zero-shot modelscited by 785
  15. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  16. 2022Large Language Models Can Self-Improvecited by 656
  17. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  18. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  19. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  20. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  21. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  22. 2019Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learningcited by 507
  23. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  24. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 483
  25. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  26. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  27. 2020Meta Pseudo Labelscited by 372
  28. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  29. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  30. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  31. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  32. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  33. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  34. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  35. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  36. 2019Billion-scale semi-supervised learning for image classificationcited by 232
  37. 2019Connecting Vision and Language with Localized Narrativescited by 225
  38. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Is user feedback always informative? Retrieval Latent Defending for Semi-Supervised Domain Adaptation without Source Datacited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025Adapting Vision-Language Models Without Labels: A Comprehensive Survey
  2. 2025Vision-Language Model Selection and Reuse for Downstream Adaptation
  3. 2024Efficient and Context-Aware Label Propagation for Zero-/Few-Shot Training-Free Adaptation of Vision-Language Model
  4. 2024How Vision-Language Tasks Benefit from Large Pre-trained Models: A Survey
  5. 2023VILA: On Pre-training for Visual Language Models
  6. 2023Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages
  7. 2023TAP: Targeted Prompting for Task Adaptive Generation of Textual Training Instances for Visual Classification
  8. 2022SVL-Adapter: Self-Supervised Adapter for Vision-Language Pretrained Models