mlsanityData to 7 Oct 2026

MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens

arXiv 2406.11271 ↗First posted 17 Jun 2024cs.CV cs.LGcited by 47

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal interleaved datasets featuring free-form interleaved sequences of images and text are crucial for training frontier large multimodal models (LMMs). Despite the rapid progression of open-source LMMs, there remains a pronounced scarcity of large-scale, diverse open-source multimodal interleaved datasets. In response, we introduce MINT-1T, the most extensive and diverse open-source Multimodal INTerleaved dataset to date. MINT-1T comprises one trillion text tokens and 3.4 billion images, a 10x scale-up from existing open-source datasets. Additionally, we include previously untapped sources such as PDFs and ArXiv papers. As scaling multimodal interleaved datasets requires substantial engineering effort, sharing the data curation process and releasing the dataset greatly benefits the community. Our experiments show that LMMs trained on MINT-1T rival the performance of models trained on the previous leading dataset, OBELICS. Our data and code will be released at https://github.com/mlfoundations/MINT-1T.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 57 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

37 of 57 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024MANTIS: Interleaved Multi-Image Instruction Tuning2 May
  2. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training14 Mar
  3. 2023Generative Multimodal Models are In-Context Learners20 Dec
  4. 2023VILA: On Pre-training for Visual Language Models12 Dec
  5. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 31

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  6. 2015VQA: Visual Question Answeringcited by 2,489
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  8. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  10. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  11. 2019Towards VQA Models That Can Readcited by 1,696
  12. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
19 more
  1. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  2. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  3. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  4. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  5. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  6. 2024Yi: Open Foundation Models by 01.AIcited by 706
  7. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  8. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  9. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  10. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  11. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  12. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  13. 2024What matters when building vision-language models?cited by 321
  14. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  15. 2023Data Filtering Networkscited by 247
  16. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  17. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  18. 2024Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Modelscited by 138
  19. 2023T-MARS: Improving Visual Representations by Circumventing Text Feature Learningcited by 43

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 21; 2026: 9.

47 papers.

Sort:

By area:

  1. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  2. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
  3. 2025MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipecited by 174
  4. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  5. 2025MMSearch-R1: Incentivizing LMMs to Searchcited by 120
  6. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
  7. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  8. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 85
  9. 2025Zebra-CoT: A Dataset for Interleaved Vision Language Reasoningcited by 78
  10. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  11. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
  12. 2024Consent in Crisis: The Rapid Decline of the AI Data Commonscited by 45
35 more
  1. 2024LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillationcited by 43
  2. 2024Baichuan-Omni Technical Reportcited by 42
  3. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  4. 2024xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMscited by 34
  5. 2024MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Modelscited by 34
  6. 2024MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Modelscited by 32
  7. 2024Group Diffusion Transformers are Unsupervised Multitask Learnerscited by 22
  8. 2024ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Modelscited by 21
  9. 2025OneFlow: Concurrent Mixed-Modal and Interleaved Generation with Edit Flowscited by 17
  10. 2025Retrieval-augmented GUI Agents with Generative Guidelinescited by 14
  11. 2025BigCharts-R1: Enhanced Chart Reasoning with Visual Reinforcement Finetuningcited by 13
  12. 2024LitLLMs, LLMs for Literature Review: Are we there yet?cited by 12
  13. 2025Visuospatial Cognitive Assistantcited by 11
  14. 2026Action100M: A Large-scale Video Action Datasetcited by 10
  15. 20252.5 Years in Class: A Multimodal Textbook for Vision-Language Pretrainingcited by 9
  16. 2024Multi-modal Generative AI: Multi-modal LLMs, Diffusions, and the Unificationcited by 8
  17. 2024BLIP3-KALE: Knowledge Augmented Large-Scale Dense Captionscited by 8
  18. 2025Generalizing vision-language models to novel domains: A comprehensive surveycited by 6
  19. 2025SMIR: Efficient Synthetic Data Pipeline To Improve Multi-Image Reasoningcited by 3
  20. 2026MultiHaystack: Benchmarking Multimodal Retrieval and Reasoning over 40K Images, Videos, and Documentscited by 3
  21. 2026Look Before Acting: Enhancing Vision Foundation Representations for Vision-Language-Action Modelscited by 3
  22. 2024Exploring Multi-Grained Concept Annotations for Multimodal Large Language Modelscited by 2
  23. 2024SynChart: Synthesizing Charts from Language Modelscited by 1
  24. 2025Should VLMs be Pre-trained with Image Data?cited by 1
  25. 2025BRIDGES: Bridging Graph Modality and Large Language Models within EDA Taskscited by 1
  26. 2025Prioritizing Image-Related Tokens Enhances Vision-Language Pre-Trainingcited by 1
  27. 2025Exploring a Unified Vision-Centric Contrastive Alternatives on Multi-Modal Web Documentscited by 1
  28. 2025Rectifying Soft-Label Entangled Bias in Long-Tailed Dataset Distillationcited by 1
  29. 2026Do LLMs and VLMs Share Neurons for Inference? Evidence and Mechanisms of Cross-Modal Transfercited by 1
  30. 2026Shaping Schema via Language Representation as the Next Frontier for LLM Intelligence Expandingcited by 1
  31. 2025Wasm: A Pipeline for Constructing Structured Arabic Interleaved Multimodal Corporacited by 0
  32. 2026Chitrakshara: A Large Multilingual Multimodal Dataset for Indian languagescited by 0
  33. 2026EXAONE 4.5 Technical Reportcited by 0
  34. 2026MultAttnAttrib: Training-Free Multimodal Attribution in Long Document Question Answeringcited by 0
  35. 2026InfiMed2: A Generalist Medical Multimodal Foundation Model from Contextual Evidence and Stability-Aware Supervisioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text
  2. 2025OpenUni: A Simple Baseline for Unified Multimodal Understanding and Generation
  3. 2025DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models
  4. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents
  5. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training
  6. 2024OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation
  7. 2024Benchmarking Large Multimodal Models against Common Corruptions
  8. 2025LMM4LMM: Benchmarking and Evaluating Large-multimodal Image Generation with LMMs