mlsanityData to 13 Oct 2026

Tuning Multi-mode Token-level Prompt Alignment across Modalities

arXiv 2309.13847 ↗First posted 25 Sep 2023cs.CVcited by 26

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Advancements in prompt tuning of vision-language models have underscored their potential in enhancing open-world visual concept comprehension. However, prior works only primarily focus on single-mode (only one prompt for each modality) and holistic level (image or sentence) semantic alignment, which fails to capture the sample diversity, leading to sub-optimal prompt discovery. To address the limitation, we propose a multi-mode token-level tuning framework that leverages the optimal transportation to learn and align a set of prompt tokens across modalities. Specifically, we rely on two essential factors: 1) multi-mode prompts discovery, which guarantees diverse semantic representations, and 2) token-level alignment, which helps explore fine-grained similarity. Consequently, the similarity can be calculated as a hierarchical transportation problem between the modality-specific sets. Extensive experiments on popular image recognition benchmarks show the superior generalization and few-shot abilities of our approach. The qualitative analysis demonstrates that the learned prompt tokens have the ability to capture diverse visual concepts.

Key facts. Cited by 26 papers in the map (data through 13 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 44 of its 58 references that are in the map. First posted on arXiv 25 Sep 2023.

Builds on

44 of 58 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023Patch-Prompt Aligned Bayesian Prompt Tuning for Vision-Language Models16 Mar
  2. 2022Understanding and Mitigating Overfitting in Prompt Tuning for Vision-Language Models4 Nov
  3. 2022Prompting through Prototype: A Prototype-based Prompt Learning on Pretrained Vision-Language Models19 Oct
  4. 2022Unified Vision and Language Prompt Learning13 Oct
  5. 2022Learning to Decompose Visual Features with Latent Textual Prompts9 Oct
  6. 2022Prompt-aligned Gradient for Prompt Tuning30 May
  7. 2022Prompt Distribution Learning6 May
  8. 2022Learning to Compose Soft Prompts for Compositional Zero-Shot Learning7 Apr
  9. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  10. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2020Language Models are Few-Shot Learnerscited by 32,320
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,107
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,314
  5. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,295
  6. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,229
  7. 2013Describing Textures in the Wildcited by 2,115
  8. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,847
  9. 2013Fine-Grained Visual Classification of Aircraftcited by 1,709
  10. 2022Visual Prompt Tuningcited by 1,677
  11. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,510
  12. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,492
22 more
  1. 2019Natural Adversarial Examplescited by 1,405
  2. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,012
  3. 2019How Can We Know What Language Models Know?cited by 958
  4. 2021Learning to Prompt for Continual Learningcited by 941
  5. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  6. 2021OTA: Optimal Transport Assignment for Object Detectioncited by 177
  7. 2019Style Transfer by Relaxed Optimal Transport and Self-Similaritycited by 167
  8. 2023Explicit Visual Prompting for Low-Level Structure Segmentationscited by 114
  9. 2022Visual Prompt Tuning for Test-time Domain Adaptationcited by 97
  10. 2022Visual Prompt Tuning for Generative Transfer Learningcited by 75
  11. 2022Assaying Out-Of-Distribution Generalization in Transfer Learningcited by 53
  12. 2020Neural Topic Model via Optimal Transportcited by 50
  13. 2022Representing Mixtures of Word Embeddings with Mixtures of Topic Embeddingscited by 36
  14. 2021Evaluation of BERT and ALBERT Sentence Embedding Performance on Downstream NLP Taskscited by 32
  15. 2023POUF: Prompt-oriented unsupervised fine-tuning for large pre-trained modelscited by 29
  16. 2022Learning to Re-weight Examples with Optimal Transport for Imbalanced Classificationcited by 23
  17. 2022Prompt-Matched Semantic Segmentationcited by 22
  18. 2023Vector Quantized Wasserstein Auto-Encodercited by 22
  19. 2023PatchCT: Aligning Patch Set and Label Set with Conditional Transport for Multi-Label Image Classificationcited by 21
  20. 2021Learning Prototype-oriented Set Representations for Meta-Learningcited by 19
  21. 2022Adaptive Distribution Calibration for Few-Shot Learning with Hierarchical Optimal Transportcited by 14
  22. 2022PromptonomyViT: Multi-Task Prompt Learning Improves Video Transformers using Synthetic Scene Datacited by 14

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 1; 2024: 6; 2025: 13; 2026: 6.

26 papers.

Sort:

By area:

  1. 2023ArGue: Attribute-Guided Prompt Tuning for Vision-Language Modelscited by 39
  2. 2024AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportationcited by 27
  3. 2025Enhancing CLIP Robustness via Cross-Modality Alignmentcited by 14
  4. 2025Dynamic Multimodal Prototype Learning in Vision-Language Modelscited by 13
  5. 2025ProAPO: Progressively Automatic Prompt Optimization for Visual Classificationcited by 12
  6. 2024Efficient Test-Time Prompt Tuning for Vision-Language Modelscited by 9
  7. 2024CoMMIT: Coordinated Multimodal Instruction Tuningcited by 8
  8. 2024Tree of Attributes Prompt Learning for Vision-Language Modelscited by 7
  9. 2026Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigationcited by 7
  10. 2025Interpretable Zero-Shot Learning with Locally-Aligned Vision-Language Modelcited by 6
  11. 2025Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learningcited by 4
  12. 2025Generalizable Prompt Learning of CLIP: A Brief Overviewcited by 3
14 more
  1. 2025Neural Network Reprogrammability: A Unified Theme on Model Reprogramming, Prompt Tuning, and Prompt Instructioncited by 3
  2. 2025GLAD: Generalizable Tuning for Vision-Language Modelscited by 3
  3. 2026RETLLM: Training and Data-Free MLLMs for Multimodal Information Retrievalcited by 3
  4. 2025LLM Empowered Prototype Learning for Zero and Few-Shot Tasks on Tabular Datacited by 2
  5. 2025Language-driven Fine-grained Retrievalcited by 2
  6. 2026Improving Sparse Autoencoder with Dynamic Attentioncited by 2
  7. 2026STiTch: Semantic Transition and Transportation in Collaboration for Training-Free Zero-Shot Composed Image Retrievalcited by 2
  8. 2026GuardAlign: Test-time Safety Alignment in Multimodal Large Language Modelscited by 1
  9. 2024RESTORE: Towards Feature Shift for Vision-Language Prompt Learningcited by 0
  10. 2024Enhancing Robustness of Vision-Language Models through Orthogonality Learning and Self-Regularizationcited by 0
  11. 2025Token-Level Prompt Mixture with Parameter-Free Routing for Federated Domain Generalizationcited by 0
  12. 2025Robust Incomplete-Modality Alignment for Ophthalmic Disease Grading and Diagnosis via Labeled Optimal Transportcited by 0
  13. 2025SPANER: Shared Prompt Aligner for Multimodal Semantic Representationcited by 0
  14. 2026Zero-Shot 3D Question Answering via Hierarchical View-to-Token Transportationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Decouple before Align: Visual Disentanglement Enhances Prompt Tuning
  2. 2026Joint Semantic Token Selection and Prompt Optimization for Interpretable Prompt Learning
  3. 2024Tuning Vision-Language Models with Candidate Labels by Prompt Alignment
  4. 2023Context-Aware Prompt Tuning for Vision-Language Model with Dual-Alignment
  5. 2024Can Better Text Semantics in Prompt Tuning Improve VLM Generalization?
  6. 2024Craft: Cross-modal Aligned Features Improve Robustness of Prompt Tuning
  7. 2022Learning Domain Invariant Prompt for Vision-Language Models
  8. 2025InPK: Infusing Prior Knowledge into Prompt for Vision-Language Models