mlsanityData to 7 Oct 2026

Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Data

arXiv 2410.18558 ↗First posted 24 Oct 2024cs.CLcited by 67

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Recently, Vision-Language Models (VLMs) have achieved remarkable progress in multimodal tasks, and multimodal instruction data serves as the foundation for enhancing VLM capabilities. Despite the availability of several open-source multimodal datasets, limitations in the scale and quality of open-source instruction data hinder the performance of VLMs trained on these datasets, leading to a significant gap compared to models trained on closed-source data. To address this challenge, we introduce Infinity-MM, a large-scale multimodal instruction dataset. We collected the available multimodal instruction datasets and performed unified preprocessing, resulting in a dataset with over 40 million samples that ensures diversity and accuracy. Furthermore, to enable large-scale expansion of instruction data and support the continuous acquisition of high-quality data, we propose a synthetic instruction generation method based on a tagging system and open-source VLMs. By establishing correspondences between different types of images and associated instruction types, this method can provide essential guidance during data synthesis. Leveraging this high-quality data, we have trained a 2-billion-parameter Vision-Language Model, Aquila-VL-2B, which achieves state-of-the-art (SOTA) performance among models of similar scale. The data is available at: https://huggingface.co/datasets/BAAI/Infinity-MM.

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 51 references that are in the map. First posted on arXiv 24 Oct 2024.

Builds on

32 of 51 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  2. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  3. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phone3 Aug
  4. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models10 Jul
  5. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  6. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  7. 2023Generative Multimodal Models are In-Context Learners20 Dec
  8. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  9. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration7 Nov
  10. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  11. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  12. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
1 more
  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 19

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  4. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  5. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  6. 2016A Diagram Is Worth A Dozen Imagescited by 793
  7. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  8. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  9. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  10. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468
  11. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  12. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
7 more
  1. 2016Visual Dialogcited by 377
  2. 2021GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoningcited by 224
  3. 2023MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuningcited by 149
  4. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuningcited by 143
  5. 2020Improving Image Captioning with Better Use of Captionscited by 84
  6. 2023Open-Set Image Tagging with Multi-Grained Text Supervisioncited by 67
  7. 2017SPEECH-COCO: 600k Visually Grounded Spoken Captions Aligned to MSCOCO Data Setcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 52; 2026: 12.

67 papers.

Sort:

By area:

  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  3. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  4. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 315
  5. 2025Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learningcited by 267
  6. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  7. 2025PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understandingcited by 147
  8. 2025OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automationcited by 129
  9. 2025Emu3.5: Native Multimodal Models are World Learnerscited by 119
  10. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  11. 2025Harmonizing Visual Representations for Unified Multimodal Understanding and Generationcited by 74
  12. 2025Scalable Vision Language Model Training via High Quality Data Curationcited by 54
38 more
  1. 2025LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted Contrastive Learningcited by 54
  2. 2025Kwai Keye-VL Technical Reportcited by 47
  3. 2025EVEv2: Improved Baselines for Encoder-Free Vision-Language Modelscited by 38
  4. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  5. 2025First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Trainingcited by 34
  6. 2025Mavors: Multi-granularity Video Representation for Multimodal Large Language Modelcited by 31
  7. 2024V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encodingcited by 28
  8. 2025The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformercited by 28
  9. 2025RzenEmbed: Towards Comprehensive Multimodal Retrievalcited by 28
  10. 2025OmniAlign-V: Towards Enhanced Alignment of MLLMs with Human Preferencecited by 27
  11. 2025Valley2: Exploring Multimodal Models with Scalable Vision-Language Designcited by 20
  12. 2024LinVT: Empower Your Image-level Large Language Model to Understand Videoscited by 19
  13. 2025M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performancecited by 19
  14. 2025MagicGUI: A Foundational Mobile GUI Agent with Scalable Data Pipeline and Reinforcement Fine-tuningcited by 19
  15. 2025SATORI-R1: Incentivizing Multimodal Reasoning through Explicit Visual Anchoringcited by 17
  16. 2025SAIL-VL2 Technical Reportcited by 14
  17. 2025Visuospatial Cognitive Assistantcited by 11
  18. 20252.5 Years in Class: A Multimodal Textbook for Vision-Language Pretrainingcited by 9
  19. 2025Knowledge Bridger: Towards Training-free Missing Modality Completioncited by 9
  20. 2025GenRecal: Generation after Recalibration from Large to Small Vision-Language Modelscited by 9
  21. 2025Unified Multimodal Understanding via Byte-Pair Visual Encodingcited by 9
  22. 2025Unified Reinforcement and Imitation Learning for Vision-Language Modelscited by 8
  23. 2026LLaDA-o: An Effective and Length-Adaptive Omni Diffusion Modelcited by 8
  24. 2025Masking Teacher and Reinforcing Student for Distilling Vision-Language Modelscited by 6
  25. 2025End-to-End Vision Tokenizer Tuningcited by 5
  26. 2025Visual Instruction Bottleneck Tuningcited by 5
  27. 2025From Generator to Embedder: Harnessing Innate Abilities of Multimodal LLMs via Building Zero-Shot Discriminative Embedding Modelcited by 5
  28. 2025TokenFLEX: Unified VLM Training for Flexible Visual Tokens Inferencecited by 4
  29. 2025SAILViT: Towards Robust and Generalizable Visual Backbones for MLLMs via Gradual Feature Refinementcited by 4
  30. 2025Growing Visual Generative Capacity for Pre-Trained MLLMscited by 4
  31. 2025AndesVL Technical Report: An Efficient Mobile-side Multimodal Large Language Modelcited by 4
  32. 2026RegionReasoner: Region-Grounded Multi-Round Visual Reasoningcited by 4
  33. 2025PP-DocBee: Improving Multimodal Document Understanding Through a Bag of Trickscited by 3
  34. 2025AutoJudger: An Agent-Driven Framework for Efficient Benchmarking of MLLMscited by 3
  35. 2025AdaLRS: Loss-Guided Adaptive Learning Rate Search for Efficient Foundation Model Pretrainingcited by 3
  36. 2026Magic-MM-Embedding: Towards Visual-Token-Efficient Universal Multimodal Embedding with MLLMscited by 3
  37. 2025PlanGPT-VL: Enhancing Urban Planning with Domain-Specific Vision-Language Modelscited by 2
  38. 2025Task-Related Token Compression in Multimodal Large Language Models from an Explainability Perspectivecited by 2

50 most cited of 67 All 67 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models
  2. 2024MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct
  3. 2023Visual Instruction Tuning
  4. 2024ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Models
  5. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning
  6. 2025Advancing Multimodal LLMs by Large-Scale 3D Visual Instruction Dataset Generation
  7. 2023Text as Images: Can Multimodal Large Language Models Follow Printed Instructions in Pixels?
  8. 2023Genixer: Empowering Multimodal Large Language Models as a Powerful Data Generator