mlsanityData to 7 Oct 2026

PLOT: Prompt Learning with Optimal Transport for Vision-Language Models

arXiv 2210.01253 ↗First posted 3 Oct 2022cs.CV cs.CL cs.LGcited by 64

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

With the increasing attention to large vision-language models such as CLIP, there has been a significant amount of effort dedicated to building efficient prompts. Unlike conventional methods of only learning one single prompt, we propose to learn multiple comprehensive prompts to describe diverse characteristics of categories such as intrinsic attributes or extrinsic contexts. However, directly matching each prompt to the same visual feature is problematic, as it pushes the prompts to converge to one point. To solve this problem, we propose to apply optimal transport to match the vision and text modalities. Specifically, we first model images and the categories with visual and textual feature sets. Then, we apply a two-stage optimization strategy to learn the prompts. In the inner loop, we optimize the optimal transport distance to align visual features and prompts by the Sinkhorn algorithm, while in the outer loop, we learn the prompts by this distance from the supervised data. Extensive experiments are conducted on the few-shot recognition task and the improvement demonstrates the superiority of our method. The code is available at https://github.com/CHENGY12/PLOT.

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 69 references that are in the map. First posted on arXiv 3 Oct 2022.

Builds on

48 of 69 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022Prompt Distribution Learning6 May
  2. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  3. 2021VT-CLIP: Enhancing Vision-Language Models with Visual-guided Texts4 Dec
  4. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling6 Nov
  5. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  6. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  9. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  10. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  11. 2013Describing Textures in the Wildcited by 2,110
  12. 2019Language Models as Knowledge Bases?cited by 1,922
30 more
  1. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  2. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  3. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  4. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  5. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  6. 2019Natural Adversarial Examplescited by 1,401
  7. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  8. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  9. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  10. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  11. 2019How Can We Know What Language Models Know?cited by 953
  12. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  13. 2018Computational Optimal Transportcited by 765
  14. 2021GPT Understands, Toocited by 715
  15. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  16. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  17. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  18. 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 406
  19. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 317
  20. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  21. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  22. 2019Gromov-Wasserstein Learning for Graph Matching and Node Embeddingcited by 125
  23. 2018Improving GANs Using Optimal Transportcited by 122
  24. 2020Vocabulary Learning via Optimal Transport for Neural Machine Translationcited by 58
  25. 2020Neural Topic Model via Optimal Transportcited by 50
  26. 2021MURAL: Multimodal, Multitask Retrieval Across Languagescited by 44
  27. 2019Improving Sequence-to-Sequence Learning via Optimal Transportcited by 33
  28. 2021Towards Interpretable Deep Metric Learning with Structural Matchingcited by 30
  29. 2022Optimal transport for causal discoverycited by 19
  30. 2017Co-clustering through Optimal Transportcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 28; 2024: 19; 2025: 14; 2026: 2.

64 papers.

Sort:

By area:

  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  2. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  3. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Modelscited by 118
  4. 2023TCP:Textual-based Class-aware Prompt tuning for Visual-Language Modelcited by 75
  5. 2023A Closer Look at the Few-Shot Adaptation of Large Vision-Language Modelscited by 67
  6. 2024LP++: A Surprisingly Strong Linear Probe for Few-Shot CLIPcited by 60
  7. 2023DePT: Decoupled Prompt Tuningcited by 58
  8. 2023Improving Zero-Shot Generalization for CLIP with Synthesized Promptscited by 51
  9. 2024A Hard-to-Beat Baseline for Training-free CLIP-based Adaptationcited by 51
  10. 2024Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Modelscited by 51
  11. 2023Multi-modal Queried Object Detection in the Wildcited by 47
  12. 2024Global and Local Prompts Cooperation via Optimal Transport for Federated Learningcited by 45
38 more
  1. 2023Black Box Few-Shot Adaptation for Vision-Language modelscited by 39
  2. 2023Distribution-Aware Prompt Tuning for Vision-Language Modelscited by 35
  3. 2023Visual Tuningcited by 34
  4. 2023Generalized Logit Adjustment: Calibrating Fine-tuned Models by Removing Label Bias in Foundation Modelscited by 32
  5. 2024DiPrompT: Disentangled Prompt Tuning for Multiple Latent Domain Generalization in Federated Learningcited by 32
  6. 2023OV-PARTS: Towards Open-Vocabulary Part Segmentationcited by 31
  7. 2024Transductive Zero-Shot and Few-Shot CLIPcited by 31
  8. 2023S-CLIP: Semi-supervised Vision-Language Learning using Few Specialist Captionscited by 29
  9. 2025Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Modelscited by 23
  10. 2024Few-shot Adaptation of Medical Vision-Language Modelscited by 22
  11. 2024Historical Test-time Prompt Tuning for Vision Foundation Modelscited by 21
  12. 2023Learning to Adapt CLIP for Few-Shot Monocular Depth Estimationcited by 19
  13. 2023ZegOT: Zero-shot Segmentation Through Optimal Transport of Text Promptscited by 16
  14. 2023OST: Refining Text Knowledge with Optimal Spatio-Temporal Descriptor for General Video Recognitioncited by 16
  15. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 15
  16. 2025CLIP-Powered Domain Generalization and Domain Adaptation: A Comprehensive Surveycited by 15
  17. 2024Learning to Prompt Segment Anything Modelscited by 13
  18. 2025Learning Clustering-based Prototypes for Compositional Zero-shot Learningcited by 13
  19. 2023Patch-Prompt Aligned Bayesian Prompt Tuning for Vision-Language Modelscited by 12
  20. 2023Cross-Modal Concept Learning and Inference for Vision-Language Modelscited by 12
  21. 2023DreamDistribution: Learning Prompt Distribution for Diverse In-distribution Generationcited by 10
  22. 2024Multi-Class Textual-Inversion Secretly Yields a Semantic-Agnostic Classifiercited by 10
  23. 2024OTSeg: Multi-prompt Sinkhorn Attention for Zero-Shot Semantic Segmentationcited by 9
  24. 2024Efficient Test-Time Prompt Tuning for Vision-Language Modelscited by 9
  25. 2023Text Descriptions are Compressive and Invariant Representations for Visual Learningcited by 7
  26. 2023Regularized Mask Tuning: Uncovering Hidden Knowledge in Pre-trained Vision-Language Modelscited by 7
  27. 2023Open-vocabulary Video Question Answering: A New Benchmark for Evaluating the Generalizability of Video Question Answering Modelscited by 7
  28. 2024Tree of Attributes Prompt Learning for Vision-Language Modelscited by 7
  29. 2025Grad-ECLIP: Gradient-based Visual and Textual Explanations for CLIPcited by 7
  30. 2025A Closer Look at Conditional Prompt Tuning for Vision-Language Modelscited by 7
  31. 2023Consistent Optimal Transport with Empirical Conditional Measurescited by 6
  32. 2023Descriptor and Word Soups: Overcoming the Parameter Efficiency Accuracy Tradeoff for Out-of-Distribution Few-shot Learningcited by 6
  33. 2024Seek for Incantations: Towards Accurate Text-to-Image Diffusion Synthesis through Prompt Engineeringcited by 6
  34. 2024Point-PRC: A Prompt Learning Based Regulation Framework for Generalizable Point Cloud Analysiscited by 6
  35. 2023DPL: Decoupled Prompt Learning for Vision-Language Modelscited by 4
  36. 2024Revisiting Prompt Pretraining of Vision-Language Modelscited by 4
  37. 2024PLOT-TAL: Prompt Learning with Optimal Transport for Few-Shot Temporal Action Localizationcited by 3
  38. 2025Prompts Generalize with Low Data: Non-vacuous Generalization Bounds for Optimizing Prompts with More Informative Priorscited by 3

50 most cited of 64 All 64 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Dude: Dual Distribution-Aware Context Prompt Learning For Large Vision-Language Model
  2. 2025Prompt-OT: An Optimal Transport Regularization Paradigm for Knowledge Preservation in Vision-Language Model Adaptation
  3. 2023ECO: Ensembling Context Optimization for Vision-Language Models
  4. 2025ProAPO: Progressively Automatic Prompt Optimization for Visual Classification
  5. 2023PRE: Vision-Language Prompt Learning with Reparameterization Encoder
  6. 2024Learning to Prompt with Text Only Supervision for Vision-Language Models
  7. 2023Towards Robust Prompts on Vision-Language Models
  8. 2025Show or Tell? Effectively prompting Vision-Language Models for semantic segmentation