mlsanityData to 7 Oct 2026

Musketeer: Joint Training for Multi-task Vision Language Model with Task Explanation Prompts

arXiv 2305.07019 ↗First posted 11 May 2023cs.CV cs.AI cs.CLcited by 3

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

We present a vision-language model whose parameters are jointly trained on all tasks and fully shared among multiple heterogeneous tasks which may interfere with each other, resulting in a single model which we named Musketeer. The integration of knowledge across heterogeneous tasks is enabled by a novel feature called Task Explanation Prompt (TEP). With rich and structured information such as task input/output format, TEP reduces interference among tasks, allowing the model to focus on their shared structure. With a single model, Musketeer achieves results comparable to or better than strong baselines trained on single tasks, almost uniformly across multiple tasks.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 62 of its 74 references that are in the map. First posted on arXiv 11 May 2023.

Builds on

62 of 74 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Multitask Vision-Language Prompt Tuning21 Nov
  2. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling6 Nov
  3. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct

Other references · 59

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
38 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  6. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  7. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  8. 2016Modeling Context in Referring Expressionscited by 1,054
  9. 2019How Can We Know What Language Models Know?cited by 953
  10. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  11. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  12. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  13. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  14. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  15. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  16. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  17. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  18. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  19. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  20. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  21. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  22. 2013Generating Sequences With Recurrent Neural Networkscited by 549
  23. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  24. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  25. 2015A Neural Attention Model for Abstractive Sentence Summarizationcited by 394
  26. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  27. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  28. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  29. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 235
  30. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  31. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  32. 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
  33. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  34. 2021Structured Prediction as Translation between Augmented Natural Languagescited by 183
  35. 2021Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classificationcited by 183
  36. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  37. 2022A Unified Sequence Interface for Vision Taskscited by 152
  38. 2021Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Taskscited by 124

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 1.

3 papers.

  1. 2024Enhancing Vision-Language Pre-training with Rich Supervisionscited by 12
  2. 2024Non-autoregressive Sequence-to-Sequence Vision-Language Modelscited by 5
  3. 2025Task-Specific Dual-Model Framework for Comprehensive Traffic Safety Video Description and Analysiscited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2022Prompt Tuning with Soft Context Sharing for Vision-Language Models
  2. 2024ClawMachine: Learning to Fetch Visual Tokens for Referential Comprehension
  3. 2023Context-Aware Prompt Tuning for Vision-Language Model with Dual-Alignment
  4. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Models
  5. 2024Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment
  6. 2025Large (Vision) Language Models are Unsupervised In-Context Learners
  7. 2025Cross-Task Generalization Between Understanding and Generation in Unified Vision-Language Models: A Controlled Study
  8. 2026VersaViT: Enhancing MLLM Vision Backbones via Task-Guided Optimization