mlsanityData to 7 Oct 2026

Prompt Tuning with Soft Context Sharing for Vision-Language Models

arXiv 2208.13474 ↗First posted 29 Aug 2022cs.CV cs.AIcited by 13

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Vision-language models have recently shown great potential on many tasks in computer vision. Meanwhile, prior work demonstrates prompt tuning designed for vision-language models could acquire superior performance on few-shot image recognition compared to linear probe, a strong baseline. In practice, many few-shot tasks are inherently correlated, particularly within specialized domains. However, such information is overlooked previously. Inspired by the fact that modeling task relationship by multi-task learning can usually boost performance, we propose a novel method SoftCPT (Soft Context Sharing for Prompt Tuning) to tune pre-trained vision-language models on multiple target few-shot tasks jointly. Specifically, we design a task-shared meta network to generate prompt context for each task using task name together with a learnable task context as input. The parameters of this meta network as well as the task context are tuned on the joint training set of all tasks. As such, the prompt context of all tasks will be shared in a soft manner. Extensive experiments across four multi-task few-shot datasets covering 44 tasks and 1593 categories demonstrate that SoftCPT significantly outperforms single-task prompt tuning methods, highlighting the effectiveness of multi-task learning for vision-language prompt tuning. Code is available at https://github.com/kding1225/softcpt.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 65 of its 95 references that are in the map. First posted on arXiv 29 Aug 2022.

Builds on

65 of 95 references.

Closest ancestors · 7

Same direction, newest first

  1. 2022Dual Modality Prompt Tuning for Vision-Language Pre-Trained Model17 Aug
  2. 2022Prompt-aligned Gradient for Prompt Tuning30 May
  3. 2022Prompt Distribution Learning6 May
  4. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  5. 2022Learning to Compose Diversified Prompts for Image Emotion Classification26 Jan
  6. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  7. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 58

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
38 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  3. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  5. 2013Describing Textures in the Wildcited by 2,110
  6. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  7. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  8. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  9. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  10. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  11. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  12. 2019How Can We Know What Language Models Know?cited by 953
  13. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  14. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  15. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  16. 2021GPT Understands, Toocited by 715
  17. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  18. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  19. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  20. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  21. 2021Image Segmentation Using Text and Image Promptscited by 509
  22. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  23. 2020Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?cited by 435
  24. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  25. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 413
  26. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  27. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  28. 2020Multi-Task Learning with Deep Neural Networks: A Surveycited by 374
  29. 2016AID: A Benchmark Dataset for Performance Evaluation of Aerial Scene Classificationcited by 343
  30. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  31. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  32. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  33. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  34. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 253
  35. 2020AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Promptscited by 246
  36. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  37. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  38. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 223

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 5; 2024: 3; 2025: 3.

13 papers.

Sort:

By area:

  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  2. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Modelscited by 118
  3. 2023InfoPrompt: Information-Theoretic Soft Prompt Tuning for Natural Language Understandingcited by 29
  4. 2024Few-shot Adaptation of Multi-modal Foundation Models: A Surveycited by 28
  5. 2022DreamArtist++: Controllable One-Shot Text-to-Image Generation via Positive-Negative Adaptercited by 24
  6. 2025Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Modelscited by 23
  7. 2024Learning to Prompt Segment Anything Modelscited by 13
  8. 2023Deeply Coupled Cross-Modal Prompt Learningcited by 11
  9. 2023Improved Visual Fine-tuning with Natural Language Supervisioncited by 7
  10. 2023M-Tuning: Prompt Tuning with Mitigated Label Bias in Open-Set Scenarioscited by 2
  11. 2025EvoVLMA: Evolutionary Vision-Language Model Adaptationcited by 1
  12. 2025SocialFusion: Addressing Social Degradation in Pre-trained Vision-Language Modelscited by 1
1 more
  1. 2024Compositional Kronecker Context Optimization for Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Multitask Vision-Language Prompt Tuning
  2. 2023Context-Aware Prompt Tuning for Vision-Language Model with Dual-Alignment
  3. 2023Gradient-Regulated Meta-Prompt Learning for Generalizable Vision-Language Models
  4. 2022Rethinking Task Sampling for Few-shot Vision-Language Transfer Learning
  5. 2022Unified Vision and Language Prompt Learning
  6. 2025MetaTPT: Meta Test-time Prompt Tuning for Vision-Language Models
  7. 2023Prompt Tuning based Adapter for Vision-Language Model Adaption
  8. 2026Your Model Already Knows Don't Teach It, Learn to Ask It: Soft Prompting for Few-Shot Adaptation of Vision-Language Models