mlsanityData to 7 Oct 2026

M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning

arXiv 2306.04387 ↗First posted 7 Jun 2023cs.CV cs.CLcited by 126

Not on the map. Why?

Cited by 77 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Instruction tuning has significantly advanced large language models (LLMs) such as ChatGPT, enabling them to align with human instructions across diverse tasks. However, progress in open vision-language models (VLMs) has been limited due to the scarcity of high-quality instruction datasets. To tackle this challenge and promote research in the vision-language field, we introduce the Multi-Modal, Multilingual Instruction Tuning (M3IT) dataset, designed to optimize VLM alignment with human instructions. Our M3IT dataset comprises 40 carefully curated datasets, including 2.4 million instances and 400 manually written task instructions, reformatted into a vision-to-text structure. Key tasks are translated into 80 languages with an advanced translation system, ensuring broader accessibility. M3IT surpasses previous datasets regarding task coverage, instruction number and instance scale. Moreover, we develop Ying-VLM, a VLM model trained on our M3IT dataset, showcasing its potential to answer complex questions requiring world knowledge, generalize to unseen video tasks, and comprehend unseen instructions in Chinese. We have open-sourced the dataset to encourage further research.

Key facts. Cited by 126 papers in the map (data through 7 Oct 2026). Builds on 51 of its 64 references that are in the map. First posted on arXiv 7 Jun 2023.

Builds on

51 of 64 references.

References · 51

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
38 more
  1. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  2. 2019Towards VQA Models That Can Readcited by 1,696
  3. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  4. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  5. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  6. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  7. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  8. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  9. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  10. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  11. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  12. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  13. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  14. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  15. 2023Instruction Tuning with GPT-4cited by 676
  16. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  17. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  18. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  19. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  20. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  21. 2016Visual Dialogcited by 377
  22. 2015Neural Module Networkscited by 375
  23. 2019Scene Text Visual Question Answeringcited by 327
  24. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  25. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  26. 2016COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Imagescited by 235
  27. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
  28. 2016Visual Storytellingcited by 176
  29. 2021VisualMRC: Machine Reading Comprehension on Document Imagescited by 150
  30. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109
  31. 2022End-to-End Multimodal Fact-Checking and Explanation Generation: A Challenging Dataset and Modelscited by 105
  32. 2016A Hierarchical Approach for Generating Descriptive Image Paragraphscited by 104
  33. 2021e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Taskscited by 84
  34. 2015Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answeringcited by 80
  35. 2018COCO-CN for Cross-Lingual Image Tagging, Captioning and Retrievalcited by 80
  36. 2022Fengshenbang 1.0: Being the Foundation of Chinese Cognitive Intelligencecited by 70
  37. 2021MMChat: Multi-Modal Chat Dataset on Social Mediacited by 30
  38. 2017ChineseFoodNet: A large-scale Image Dataset for Chinese Food Recognitioncited by 21

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 40; 2024: 62; 2025: 19; 2026: 5.

126 papers.

Sort:

By area:

  1. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  2. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  3. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  4. 2023A Survey on Multimodal Large Language Modelscited by 898
  5. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  6. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  7. 2023A Comprehensive Overview of Large Language Modelscited by 639
  8. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  9. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  10. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  11. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  12. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
38 more
  1. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  2. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  3. 2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 344
  4. 2024What matters when building vision-language models?cited by 321
  5. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 314
  6. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  7. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  8. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  9. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 197
  10. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  11. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  12. 2023SVIT: Scaling up Visual Instruction Tuningcited by 161
  13. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  14. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 132
  15. 2023Silkie: Preference Distillation for Large Visual Language Modelscited by 128
  16. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
  17. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuningcited by 106
  18. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Usecited by 100
  19. 2023Evaluation and Analysis of Hallucination in Large Vision-Language Modelscited by 99
  20. 2023AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Modelcited by 93
  21. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 85
  22. 2024MMToM-QA: Multimodal Theory of Mind Question Answeringcited by 84
  23. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMscited by 84
  24. 2024DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Modelscited by 74
  25. 2024E.T. Bench: Towards Open-Ended Event-Level Video-Language Understandingcited by 68
  26. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directionscited by 66
  27. 2023Making Large Language Models Better Reasoners with Alignmentcited by 63
  28. 2024Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languagescited by 63
  29. 2023Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Modelscited by 59
  30. 2023Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languagescited by 52
  31. 2023Large Language Models are Temporal and Causal Reasoners for Video Question Answeringcited by 51
  32. 2023Towards End-to-End Embodied Decision Making via Multi-modal Large Language Model: Explorations with GPT4-Vision and Beyondcited by 50
  33. 2024ImgTrojan: Jailbreaking Vision-Language Models with ONE Imagecited by 50
  34. 2025UniToken: Harmonizing Multimodal Understanding and Generation through Unified Visual Encodingcited by 49
  35. 2023One-Shot Learning as Instruction Data Prospector for Large Language Modelscited by 43
  36. 2023Visual Instruction Tuning with Polite Flamingocited by 38
  37. 2024MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversitycited by 38
  38. 2023StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Datacited by 33

50 most cited of 126 All 126 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuning
  2. 2024MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Models
  3. 2024Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning
  4. 2025Better Reasoning with Less Data: Enhancing VLMs Through Unified Modality Scoring
  5. 2026Empowering Reliable Visual-Centric Instruction Following in MLLMs
  6. 2024Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Data
  7. 2023Reformulating Vision-Language Foundation Models and Datasets Towards Universal Multimodal Assistants
  8. 2024Less is More: High-value Data Selection for Visual Instruction Tuning