mlsanityData to 7 Oct 2026

Prompt Learning via Meta-Regularization

arXiv 2404.00851 ↗First posted 1 Apr 2024cs.CVcited by 26

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Pre-trained vision-language models have shown impressive success on various computer vision tasks with their zero-shot generalizability. Recently, prompt learning approaches have been explored to efficiently and effectively adapt the vision-language models to a variety of downstream tasks. However, most existing prompt learning methods suffer from task overfitting since the general knowledge of the pre-trained vision language models is forgotten while the prompts are finetuned on a small data set from a specific target task. To address this issue, we propose a Prompt Meta-Regularization (ProMetaR) to improve the generalizability of prompt learning for vision-language models. Specifically, ProMetaR meta-learns both the regularizer and the soft prompts to harness the task-specific knowledge from the downstream tasks and task-agnostic general knowledge from the vision-language models. Further, ProMetaR augments the task to generate multiple virtual tasks to alleviate the meta-overfitting. In addition, we provide the analysis to comprehend how ProMetaR improves the generalizability of prompt tuning in the perspective of the gradient alignment. Our extensive experiments demonstrate that our ProMetaR improves the generalizability of conventional prompt learning methods under base-to-base/base-to-new and domain generalization settings. The code of ProMetaR is available at https://github.com/mlvlab/ProMetaR.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 65 of its 87 references that are in the map. First posted on arXiv 1 Apr 2024.

Builds on

65 of 87 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Distribution-Aware Prompt Tuning for Vision-Language Models6 Sep
  2. 2023Read-only Prompt Optimization for Vision-Language Few-shot Learning29 Aug
  3. 2023Gradient-Regulated Meta-Prompt Learning for Generalizable Vision-Language Models12 Mar
  4. 2022Unified Vision and Language Prompt Learning13 Oct
  5. 2022Prompt-aligned Gradient for Prompt Tuning30 May
  6. 2022Prompt Distribution Learning6 May
  7. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  8. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 57

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  5. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  10. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  11. 2016Matching Networks for One Shot Learningcited by 2,246
  12. 2013Describing Textures in the Wildcited by 2,110
38 more
  1. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  2. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  3. 2022Visual Prompt Tuningcited by 1,674
  4. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  5. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  6. 2019Natural Adversarial Examplescited by 1,401
  7. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  8. 2017Learning to Compare: Relation Network for Few-Shot Learningcited by 1,139
  9. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  10. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  11. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  12. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  13. 2021Robust fine-tuning of zero-shot modelscited by 785
  14. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  15. 2017Learning to Generalize: Meta-Learning for Domain Generalizationcited by 762
  16. 2021Domain Generalization: A Surveycited by 760
  17. 2021GPT Understands, Toocited by 715
  18. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  19. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  20. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  21. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  22. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  23. 2021Image Segmentation Using Text and Image Promptscited by 509
  24. 2019Meta-Learning with Differentiable Convex Optimizationcited by 461
  25. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  26. 2018How to train your MAMLcited by 306
  27. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  28. 2017Meta Networkscited by 268
  29. 2017Zero-Shot Learning -- The Good, the Bad and the Uglycited by 254
  30. 2022Patching open-vocabulary models by interpolating weightscited by 187
  31. 2018Recasting Gradient-Based Meta-Learning as Hierarchical Bayescited by 180
  32. 2020Domain Adaptive Ensemble Learningcited by 156
  33. 2020SaliencyMix: A Saliency Guided Data Augmentation Strategy for Better Regularizationcited by 149
  34. 2022PromptDet: Towards Open-vocabulary Detection using Uncurated Imagescited by 133
  35. 2021Co-Mixup: Saliency Guided Joint Mixup with Supermodular Diversitycited by 127
  36. 2018Three Mechanisms of Weight Decay Regularizationcited by 112
  37. 2019Meta-Learning via Learned Losscited by 60
  38. 2021Meta-Learning with Fewer Tasks through Task Interpolationcited by 45

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 17; 2026: 2.

26 papers.

Sort:

By area:

  1. 2025X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Modelcited by 285
  2. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 15
  3. 2025Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classificationcited by 14
  4. 2025Adaptive Multi-prompt Contrastive Network for Few-shot Out-of-distribution Detectioncited by 8
  5. 2025Learning to Generalize without Bias for Open-Vocabulary Action Recognitioncited by 7
  6. 2026MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptationcited by 5
  7. 2024Few-Shot Image Quality Assessment via Adaptation of Vision-Language Modelscited by 4
  8. 2025Multimodal Prompt Alignment for Facial Expression Recognitioncited by 4
  9. 2025Generalizable Prompt Learning of CLIP: A Brief Overviewcited by 3
  10. 2025Visual Diversity and Region-aware Prompt Learning for Zero-shot HOI Detectioncited by 3
  11. 2025Modality Alignment across Trees on Heterogeneous Hyperbolic Manifoldscited by 3
  12. 2025VaMP: Variational Multi-Modal Prompt Learning for Vision-Language Modelscited by 3
14 more
  1. 2024Active Prompt Learning with Vision-Language Model Priorscited by 2
  2. 2025MetaWriter: Personalized Handwritten Text Recognition Using Meta-Learned Prompt Tuningcited by 2
  3. 2025Multi-modal Mutual-Guidance Conditional Prompt Learning for Vision-Language Modelscited by 2
  4. 2025Causality-guided Prompt Learning for Vision-language Models via Visual Granulationcited by 2
  5. 2024Exploring Text-Guided Single Image Editing for Remote Sensing Imagescited by 1
  6. 2024CoAPT: Context Attribute words for Prompt Tuningcited by 1
  7. 2024Craft: Cross-modal Aligned Features Improve Robustness of Prompt Tuningcited by 1
  8. 2025Beyond the Seen: Bounded Distribution Estimation for Open-Vocabulary Learningcited by 1
  9. 2025MetaTPT: Meta Test-time Prompt Tuning for Vision-Language Modelscited by 1
  10. 2024E2MPL:An Enduring and Efficient Meta Prompt Learning Framework for Few-shot Unsupervised Domain Adaptationcited by 0
  11. 2024Enhancing Robustness of Vision-Language Models through Orthogonality Learning and Self-Regularizationcited by 0
  12. 2025InPK: Infusing Prior Knowledge into Prompt for Vision-Language Modelscited by 0
  13. 2025Decoupling Augmentation Bias in Prompt Learning for Vision-Language Modelscited by 0
  14. 2026AdaBoosting Text Prompts for Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Prompt-OT: An Optimal Transport Regularization Paradigm for Knowledge Preservation in Vision-Language Model Adaptation
  2. 2024Revisiting Prompt Pretraining of Vision-Language Models
  3. 2023Debiased Fine-Tuning for Vision-language Models by Prompt Regularization
  4. 2022Learning Domain Invariant Prompt for Vision-Language Models
  5. 2023PRE: Vision-Language Prompt Learning with Reparameterization Encoder
  6. 2023Context-Aware Prompt Tuning for Vision-Language Model with Dual-Alignment
  7. 2022Multitask Vision-Language Prompt Tuning
  8. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models