mlsanityData to 7 Oct 2026

MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning

arXiv 2212.10773 ↗First posted 21 Dec 2022cs.CLcited by 109

Not on the map. Why?

Cited by 45 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Instruction tuning, a new learning paradigm that fine-tunes pre-trained language models on tasks specified through instructions, has shown promising zero-shot performance on various natural language processing tasks. However, it has yet to be explored for vision and multimodal tasks. In this work, we introduce MUL-TIINSTRUCT, the first multimodal instruction tuning benchmark dataset that consists of 62 diverse multimodal tasks in a unified seq-to-seq format covering 10 broad categories. The tasks are derived from 21 existing open-source datasets and each task is equipped with 5 expert-written instructions. We take OFA as the base pre-trained model for multimodal instruction tuning, and to further improve its zero-shot performance, we explore multiple transfer learning strategies to leverage the large-scale NATURAL INSTRUCTIONS dataset. Experimental results demonstrate strong zero-shot performance on various unseen multimodal tasks and the benefit of transfer learning from a text-only instruction dataset. We also design a new evaluation metric - Sensitivity, to evaluate how sensitive the model is to the variety of instructions. Our results indicate that fine-tuning the model on a diverse set of tasks and instructions leads to a reduced sensitivity to variations in instructions for each task.

Key facts. Cited by 109 papers in the map (data through 7 Oct 2026). Builds on 49 of its 66 references that are in the map. First posted on arXiv 21 Dec 2022.

Builds on

49 of 66 references.

References · 49

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  12. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
37 more
  1. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  2. 2019Towards VQA Models That Can Readcited by 1,696
  3. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  4. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  5. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  6. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  7. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  8. 2016Modeling Context in Referring Expressionscited by 1,054
  9. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  10. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  11. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  12. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  13. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  14. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  15. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  16. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  17. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  18. 2021MetaICL: Learning to Learn In Contextcited by 450
  19. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  20. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  21. 2016Visual Dialogcited by 377
  22. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  23. 2015Visual7W: Grounded Question Answering in Imagescited by 315
  24. 2022Visual Spatial Reasoningcited by 288
  25. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  26. 2021PTR: Prompt Tuning with Rules for Text Classificationcited by 238
  27. 2016COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Imagescited by 235
  28. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  29. 2022End-to-End Multimodal Fact-Checking and Explanation Generation: A Challenging Dataset and Modelscited by 105
  30. 2021Learning to Predict Visual Attributes in the Wildcited by 95
  31. 2017An Analysis of Visual Question Answering Algorithmscited by 90
  32. 2022LAVENDER: Unifying Video-Language Understanding as Masked Language Modelingcited by 73
  33. 2022Uni-Perceiver v2: A Generalist Model for Large-Scale Vision and Vision-Language Taskscited by 57
  34. 2022End-to-end Spoken Conversational Question Answering: Task, Dataset and Modelcited by 37
  35. 2020Assessing Image Quality Issues for Real-World Problemscited by 28
  36. 2021MEDIC: A Multi-Task Learning Dataset for Disaster Image Classificationcited by 20
  37. 2022The Art of Prompting: Event Detection based on Type Specific Promptscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 56; 2024: 37; 2025: 13; 2026: 2.

109 papers.

Sort:

By area:

  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  2. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  3. 2023A Survey on Multimodal Large Language Modelscited by 898
  4. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  5. 2022A Survey on In-context Learningcited by 717
  6. 2023A Comprehensive Overview of Large Language Modelscited by 639
  7. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  8. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  9. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  10. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  11. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  12. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
38 more
  1. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  2. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  3. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
  4. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  5. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  6. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  7. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 168
  8. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 165
  9. 2024Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Modelscited by 138
  10. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 133
  11. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  12. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125
  13. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMscited by 114
  14. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Usecited by 100
  15. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedbackcited by 99
  16. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directionscited by 66
  17. 2023ChatBridge: Bridging Modalities with Large Language Model as a Language Catalystcited by 64
  18. 2024CogCoM: A Visual Language Model with Chain-of-Manipulations Reasoningcited by 64
  19. 2023Evaluating the Zero-shot Robustness of Instruction-tuned Language Modelscited by 63
  20. 2023Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewritescited by 61
  21. 2023Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Modelscited by 59
  22. 2024Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Modelscited by 56
  23. 2024What Makes Multimodal In-Context Learning Work?cited by 54
  24. 2025FineVision: Open Data Is All You Needcited by 51
  25. 2023Improving Diffusion Models for Scene Text Editing with Dual Encoderscited by 44
  26. 2023UnIVAL: Unified Model for Image, Video, Audio and Language Taskscited by 44
  27. 2024eCeLLM: Generalizing Large Language Models for E-commerce from Large-scale, High-quality Instruction Datacited by 44
  28. 2023Digital Life Project: Autonomous 3D Characters with Social Intelligencecited by 43
  29. 2024Multimodal Table Understandingcited by 43
  30. 2024EmoVIT: Revolutionizing Emotion Insights with Visual Instruction Tuningcited by 42
  31. 2025MM-IFEngine: Towards Multimodal Instruction Followingcited by 41
  32. 2024MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversitycited by 38
  33. 2023GPT4Video: A Unified Multimodal Large Language Model for lnstruction-Followed Understanding and Safety-Aware Generationcited by 36
  34. 2024Peacock: A Family of Arabic Multimodal Large Language Models and Benchmarkscited by 32
  35. 2023Large Language Model Instruction Following: A Survey of Progresses and Challengescited by 31
  36. 2023Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learningcited by 30
  37. 2023MUFFIN: Curating Multi-Faceted Instructions for Improving Instruction-Followingcited by 30
  38. 2024RSUniVLM: A Unified Vision Language Model for Remote Sensing via Granularity-oriented Mixture of Expertscited by 29

50 most cited of 109 All 109 (JSON)