mlsanityData to 7 Oct 2026

A Retrospect to Multi-prompt Learning across Vision and Language

arXiv 2511.00191 ↗First posted 31 Oct 2025cs.CV cs.AI cs.LGcited by 2

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

The vision community is undergoing the unprecedented progress with the emergence of Vision-Language Pretraining Models (VLMs). Prompt learning plays as the holy grail of accessing VLMs since it enables their fast adaptation to downstream tasks with limited resources. Whereas existing researches milling around single-prompt paradigms, rarely investigate the technical potential behind their multi-prompt learning counterparts. This paper aims to provide a principled retrospect for vision-language multi-prompt learning. We extend the recent constant modality gap phenomenon to learnable prompts and then, justify the superiority of vision-language transfer with multi-prompt augmentation, empirically and theoretically. In terms of this observation, we propose an Energy-based Multi-prompt Learning (EMPL) to generate multiple prompt embeddings by drawing instances from an energy-based distribution, which is implicitly defined by VLMs. So our EMPL is not only parameter-efficient but also rigorously lead to the balance between in-domain and out-of-domain open-vocabulary generalization. Comprehensive experiments have been conducted to justify our claims and the excellence of EMPL.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 62 references that are in the map. First posted on arXiv 31 Oct 2025.

Builds on

43 of 62 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022PLOT: Prompt Learning with Optimal Transport for Vision-Language Models3 Oct
  2. 2022Prompt Distribution Learning6 May
  3. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  4. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  10. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2013Describing Textures in the Wildcited by 2,110
27 more
  1. 2019Language Models as Knowledge Bases?cited by 1,922
  2. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  3. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  4. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  5. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  6. 2019Natural Adversarial Examplescited by 1,401
  7. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  8. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  9. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  10. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  11. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  12. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  13. 2021Image Segmentation Using Text and Image Promptscited by 509
  14. 2019Your Classifier is Secretly an Energy Based Model and You Should Treat it Like Onecited by 402
  15. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  16. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  17. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 271
  18. 2017Zero-Shot Learning -- The Good, the Bad and the Uglycited by 254
  19. 2022Domain Adaptation via Prompt Learningcited by 128
  20. 2020Learning Latent Space Energy-Based Prior Modelcited by 104
  21. 2020Graph Optimal Transport for Cross-Domain Alignmentcited by 103
  22. 2021CrossCLR: Cross-modal Contrastive Learning For Multi-modal Video Representationscited by 72
  23. 2023Diagnosing and Rectifying Vision Models using Languagecited by 58
  24. 2022Towards Open-vocabulary Scene Graph Generation with Prompt-based Finetuningcited by 54
  25. 2021Energy-Based Open-World Uncertainty Modeling for Confidence Calibrationcited by 46
  26. 2016Deep Directed Generative Models with Energy-Based Probability Estimationcited by 34
  27. 2022Open-Vocabulary Multi-Label Classification via Multi-Modal Knowledge Transfercited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025AmPLe: Supporting Vision-Language Models via Adaptive-Debiased Ensemble Multi-Prompt Learningcited by 0
  2. 2026Federated Joint Learning for Domain and Class Generalizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MuAP: Multi-step Adaptive Prompt Learning for Vision-Language Model with Missing Modality
  2. 2023CoPL: Contextual Prompt Learning for Vision-Language Understanding
  3. 2024Revisiting Prompt Pretraining of Vision-Language Models
  4. 2022Multitask Vision-Language Prompt Tuning
  5. 2026MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptation
  6. 2022Learning Domain Invariant Prompt for Vision-Language Models
  7. 2025Prompt-OT: An Optimal Transport Regularization Paradigm for Knowledge Preservation in Vision-Language Model Adaptation
  8. 2024Optimization of Prompt Learning via Multi-Knowledge Representation for Vision-Language Models