# Prompt Learning via Meta-Regularization

arXiv: https://arxiv.org/abs/2404.00851 · HTML: https://mlsanity.com/paper/2404.00851/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 65 of its 87 references that are in the map.
- First posted on arXiv 1 Apr 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 30 of 65)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)
- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)
- [LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention](https://mlsanity.com/paper/2303.16199/) (2023)
- [Domain Generalization: A Survey](https://mlsanity.com/paper/2103.02503/) (2021)
- [Learning to Generalize: Meta-Learning for Domain Generalization](https://mlsanity.com/paper/1710.03463/) (2017)

## Built on by (most cited first, 26 of 26)

- [X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Model](https://mlsanity.com/paper/2510.10274/) (2025)
- [Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Model](https://mlsanity.com/paper/2503.04543/) (2025)
- [Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classification](https://mlsanity.com/paper/2505.01660/) (2025)
- [Adaptive Multi-prompt Contrastive Network for Few-shot Out-of-distribution Detection](https://mlsanity.com/paper/2506.17633/) (2025)
- [Learning to Generalize without Bias for Open-Vocabulary Action Recognition](https://mlsanity.com/paper/2502.20158/) (2025)
- [MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptation](https://mlsanity.com/paper/2602.21397/) (2026)
- [Few-Shot Image Quality Assessment via Adaptation of Vision-Language Models](https://mlsanity.com/paper/2409.05381/) (2024)
- [Multimodal Prompt Alignment for Facial Expression Recognition](https://mlsanity.com/paper/2506.21017/) (2025)
- [Generalizable Prompt Learning of CLIP: A Brief Overview](https://mlsanity.com/paper/2503.01263/) (2025)
- [Visual Diversity and Region-aware Prompt Learning for Zero-shot HOI Detection](https://mlsanity.com/paper/2510.25094/) (2025)
- [Modality Alignment across Trees on Heterogeneous Hyperbolic Manifolds](https://mlsanity.com/paper/2510.27391/) (2025)
- [VaMP: Variational Multi-Modal Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2511.22664/) (2025)
- [Active Prompt Learning with Vision-Language Model Priors](https://mlsanity.com/paper/2411.16722/) (2024)
- [MetaWriter: Personalized Handwritten Text Recognition Using Meta-Learned Prompt Tuning](https://mlsanity.com/paper/2505.20513/) (2025)
- [Multi-modal Mutual-Guidance Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2507.08410/) (2025)
- [Causality-guided Prompt Learning for Vision-language Models via Visual Granulation](https://mlsanity.com/paper/2509.03803/) (2025)
- [Exploring Text-Guided Single Image Editing for Remote Sensing Images](https://mlsanity.com/paper/2405.05769/) (2024)
- [CoAPT: Context Attribute words for Prompt Tuning](https://mlsanity.com/paper/2407.13808/) (2024)
- [Craft: Cross-modal Aligned Features Improve Robustness of Prompt Tuning](https://mlsanity.com/paper/2407.15894/) (2024)
- [Beyond the Seen: Bounded Distribution Estimation for Open-Vocabulary Learning](https://mlsanity.com/paper/2510.04770/) (2025)
- [MetaTPT: Meta Test-time Prompt Tuning for Vision-Language Models](https://mlsanity.com/paper/2512.12268/) (2025)
- [E2MPL:An Enduring and Efficient Meta Prompt Learning Framework for Few-shot Unsupervised Domain Adaptation](https://mlsanity.com/paper/2407.04066/) (2024)
- [Enhancing Robustness of Vision-Language Models through Orthogonality Learning and Self-Regularization](https://mlsanity.com/paper/2407.08374/) (2024)
- [InPK: Infusing Prior Knowledge into Prompt for Vision-Language Models](https://mlsanity.com/paper/2502.19777/) (2025)
- [Decoupling Augmentation Bias in Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2511.03367/) (2025)
- [AdaBoosting Text Prompts for Vision-Language Models](https://mlsanity.com/paper/2607.00684/) (2026)

## Abstract (arXiv)

Pre-trained vision-language models have shown impressive success on various computer vision tasks with their zero-shot generalizability. Recently, prompt learning approaches have been explored to efficiently and effectively adapt the vision-language models to a variety of downstream tasks. However, most existing prompt learning methods suffer from task overfitting since the general knowledge of the pre-trained vision language models is forgotten while the prompts are finetuned on a small data set from a specific target task. To address this issue, we propose a Prompt Meta-Regularization (ProMetaR) to improve the generalizability of prompt learning for vision-language models. Specifically, ProMetaR meta-learns both the regularizer and the soft prompts to harness the task-specific knowledge from the downstream tasks and task-agnostic general knowledge from the vision-language models. Further, ProMetaR augments the task to generate multiple virtual tasks to alleviate the meta-overfitting. In addition, we provide the analysis to comprehend how ProMetaR improves the generalizability of prompt tuning in the perspective of the gradient alignment. Our extensive experiments demonstrate that our ProMetaR improves the generalizability of conventional prompt learning methods under base-to-base/base-to-new and domain generalization settings. The code of ProMetaR is available at https://github.com/mlvlab/ProMetaR.