Data to 7 Oct 2026

Prompt-aligned Gradient for Prompt Tuning

arXiv 2205.14865 ↗First posted 30 May 2022cs.CVcited by 339

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Thanks to the large pre-trained vision-language models (VLMs) like CLIP, we can craft a zero-shot classifier by "prompt", e.g., the confidence score of an image being "[CLASS]" can be obtained by using the VLM provided similarity measure between the image and the prompt sentence "a photo of a [CLASS]". Therefore, prompt shows a great potential for fast adaptation of VLMs to downstream tasks if we fine-tune the prompt-based similarity measure. However, we find a common failure that improper fine-tuning may not only undermine the prompt's inherent prediction for the task-related classes, but also for other classes in the VLM vocabulary. Existing methods still address this problem by using traditional anti-overfitting techniques such as early stopping and data augmentation, which lack a principled solution specific to prompt. We present Prompt-aligned Gradient, dubbed ProGrad, to prevent prompt tuning from forgetting the the general knowledge learned from VLMs. In particular, ProGrad only updates the prompt whose gradient is aligned (or non-conflicting) to the "general direction", which is represented as the gradient of the KL loss of the pre-defined prompt prediction. Extensive experiments demonstrate the stronger few-shot generalization ability of ProGrad over state-of-the-art prompt tuning methods. Codes are available at https://github.com/BeierZhu/Prompt-align.

Key facts. Cited by 339 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 70 references that are in the map. First posted on arXiv 30 May 2022.

Builds on

47 of 70 references.

Closest ancestors · 6

Same direction, newest first

Other references · 41

2015Deep Residual Learning for Image Recognitioncited by 51,809
2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
2015Distilling the Knowledge in a Neural Networkcited by 10,076
2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
2016Learning without Forgettingcited by 2,465
2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
2013Describing Textures in the Wildcited by 2,110
2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842

29 more

2013Fine-Grained Visual Classification of Aircraftcited by 1,701
2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
2019Natural Adversarial Examplescited by 1,401
2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
2020Gradient Surgery for Multi-Task Learningcited by 1,210
2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
2018Multi-Task Learning as Multi-Objective Optimizationcited by 809
2021GPT Understands, Toocited by 715
2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 511
2019Orthogonal Gradient Descent for Continual Learningcited by 361
2020Counterfactual VQA: A Cause-Effect Look at Language Biascited by 266
2021Gradient Matching for Domain Generalizationcited by 258
2020Mnemonics Training: Multi-Class Incremental Learning without Forgettingcited by 220
2019Learning De-biased Representations with Biased Representationscited by 199
2021Fishr: Invariant Gradient Variances for Out-of-Distribution Generalizationcited by 186
2021Towards Understanding Knowledge Distillationcited by 158
2021Distilling Causal Effect of Data in Class-Incremental Learningcited by 117
2021LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5cited by 103
2021WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Trainingcited by 84
2013Generalization Bounds for Domain Adaptationcited by 27
2021Introspective Distillation for Robust Question Answeringcited by 25
2021Cross-Domain Empirical Risk Minimization for Unbiased Long-tailed Classificationcited by 19
2022Continual Few-shot Relation Learning via Embedding Space Regularization and Data Augmentationcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 10; 2023: 54; 2024: 122; 2025: 108; 2026: 45.

339 papers.

Sort:

By area:

2023Vision-Language Models for Vision Tasks: A Surveycited by 834
2022MaPLe: Multi-modal Prompt Learningcited by 685
2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimizationcited by 307
2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
2024Low-Rank Few-Shot Adaptation of Vision-Language Modelscited by 103
2023Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Modelscited by 91
2023Consistency-guided Prompt Learning for Vision-Language Modelscited by 73
2024Unlearning Concepts in Diffusion Model via Concept Domain Correction and Concept Preserving Gradientcited by 69
2023A Closer Look at the Few-Shot Adaptation of Large Vision-Language Modelscited by 67
2024On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?cited by 65
2023Exploring Diverse In-Context Configurations for Image Captioningcited by 63

38 more

2023BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIPcited by 63
2024LP++: A Surprisingly Strong Linear Probe for Few-Shot CLIPcited by 60
2022LASP: Text-to-Text Optimization for Language-Aware Soft Prompting of Vision & Language Modelscited by 58
2023DePT: Decoupled Prompt Tuningcited by 58
2024Improved Zero-Shot Classification by Adapting VLMs with Text Descriptionscited by 52
2022Dual Modality Prompt Tuning for Vision-Language Pre-Trained Modelcited by 51
2022Bayesian Prompt Learning for Image-Language Model Generalizationcited by 51
2022Texts as Images in Prompt Tuning for Multi-Label Image Recognitioncited by 51
2024Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Modelscited by 51
2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
2022Understanding and Mitigating Overfitting in Prompt Tuning for Vision-Language Modelscited by 40
2023Black Box Few-Shot Adaptation for Vision-Language modelscited by 39
2024Few-Shot Adversarial Prompt Learning on Vision-Language Modelscited by 39
2023SD4Match: Learning to Prompt Stable Diffusion Model for Semantic Matchingcited by 38
2023StyLIP: Multi-Scale Style-Conditioned Prompt Learning for CLIP-based Domain Generalizationcited by 36
2024Boosting Vision-Language Models with Transductioncited by 36
2024Dual Prototype Evolving for Test-Time Generalization of Vision-Language Modelscited by 36
2023Progressive Visual Prompt Learning with Contrastive Feature Re-formationcited by 35
2022SgVA-CLIP: Semantic-guided Visual Adapting of Vision-Language Models for Few-shot Image Classificationcited by 34
2023Chain of Thought Prompt Tuning in Vision Language Modelscited by 34
2023Visual Tuningcited by 34
20245%>100%: Breaking Performance Shackles of Full Fine-Tuning on Visual Recognition Taskscited by 34
2022FedTune: A Deep Dive into Efficient Federated Fine-Tuning with Pre-trained Transformerscited by 33
2023Generalized Logit Adjustment: Calibrating Fine-tuned Models by Removing Label Bias in Foundation Modelscited by 32
2023Revisiting Few-Shot Object Detection with Vision-Language Modelscited by 32
2024Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Modelscited by 30
2023LoGoPrompt: Synthetic Text Images Can Be Good Visual Prompts for Vision-Language Modelscited by 29
2025Visual Instance-aware Prompt Tuningcited by 29
2024Few-shot Adaptation of Multi-modal Foundation Models: A Surveycited by 28
2024Robust Calibration of Large Vision-Language Adapterscited by 28
2024BiomedCoOp: Learning to Prompt for Biomedical Vision-Language Modelscited by 28
2024AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportationcited by 27
2023Tuning Multi-mode Token-level Prompt Alignment across Modalitiescited by 26
2024Prompt Learning via Meta-Regularizationcited by 26
2024Harmonizing Generalization and Personalization in Federated Prompt Learningcited by 26
2024Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correctingcited by 26
2023APPLeNet: Visual Attention Parameterized Prompt Learning for Few-Shot Remote Sensing Image Generalization using CLIPcited by 24
2024Any-Shift Prompting for Generalization over Distributionscited by 24

50 most cited of 339 All 339 (JSON)

Similar, not cited

Close in wording, no citation either way