mlsanityData to 7 Oct 2026

LaViP:Language-Grounded Visual Prompts

arXiv 2312.10945 ↗First posted 18 Dec 2023cs.CV cs.CL cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

We introduce a language-grounded visual prompting method to adapt the visual encoder of vision-language models for downstream tasks. By capitalizing on language integration, we devise a parameter-efficient strategy to adjust the input of the visual encoder, eliminating the need to modify or add to the model's parameters. Due to this design choice, our algorithm can operate even in black-box scenarios, showcasing adaptability in situations where access to the model's parameters is constrained. We will empirically demonstrate that, compared to prior art, grounding visual prompts with language enhances both the accuracy and speed of adaptation. Moreover, our algorithm excels in base-to-novel class generalization, overcoming limitations of visual prompting and exhibiting the capacity to generalize beyond seen classes. We thoroughly assess and evaluate our method across a variety of image recognition datasets, such as EuroSAT, UCF101, DTD, and CLEVR, spanning different learning situations, including few-shot learning, base-to-novel class generalization, and transfer learning.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 57 references that are in the map. First posted on arXiv 18 Dec 2023.

Builds on

31 of 57 references.

References · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Segment Anythingcited by 8,158
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2013Describing Textures in the Wildcited by 2,110
  10. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  11. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  12. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
19 more
  1. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  2. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  3. 2021Robust fine-tuning of zero-shot modelscited by 785
  4. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  5. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  6. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  7. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  8. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 271
  9. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  10. 2018Adversarial Reprogramming of Neural Networkscited by 115
  11. 2021Voice2Series: Reprogramming Acoustic Models for Time Series Classificationcited by 91
  12. 2023Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Modelscited by 91
  13. 2022Understanding and Improving Visual Prompting: A Label-Mapping Perspectivecited by 86
  14. 2020Transfer Learning without Knowing: Reprogramming Black-box Machine Learning Models with Scarce Data and Limited Resourcescited by 83
  15. 2023BlackVIP: Black-Box Visual Prompting for Robust Transfer Learningcited by 70
  16. 2022Using Language to Extend to Unseen Domainscited by 34
  17. 2022AdaPrompt: Adaptive Model Training for Prompt-based NLPcited by 30
  18. 2020Reprogramming Language Models for Molecular Representation Learningcited by 13
  19. 2020Kronecker Attention Networkscited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025A Closer Look at Conditional Prompt Tuning for Vision-Language Models
  2. 2025InPK: Infusing Prior Knowledge into Prompt for Vision-Language Models
  3. 2024Revisiting Prompt Pretraining of Vision-Language Models
  4. 2023CoPL: Contextual Prompt Learning for Vision-Language Understanding
  5. 2023Knowledge-Aware Prompt Tuning for Generalizable Vision-Language Models
  6. 2024Learning to Prompt with Text Only Supervision for Vision-Language Models
  7. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models
  8. 2025ProAPO: Progressively Automatic Prompt Optimization for Visual Classification