mlsanityData to 7 Oct 2026

OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text

arXiv 2406.08418 ↗First posted 12 Jun 2024cs.CV cs.AIcited by 58

Not on the map. Why?

Cited by 35 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Image-text interleaved data, consisting of multiple images and texts arranged in a natural document format, aligns with the presentation paradigm of internet data and closely resembles human reading habits. Recent studies have shown that such data aids multimodal in-context learning and maintains the capabilities of large language models during multimodal fine-tuning. However, the limited scale and diversity of current image-text interleaved data restrict the development of multimodal large language models. In this paper, we introduce OmniCorpus, a 10 billion-scale image-text interleaved dataset. Using an efficient data engine, we filter and extract large-scale high-quality documents, which contain 8.6 billion images and 1,696 billion text tokens. Compared to counterparts (e.g., MMC4, OBELICS), our dataset 1) has 15 times larger scales while maintaining good data quality; 2) features more diverse sources, including both English and non-English websites as well as video-centric websites; 3) is more flexible, easily degradable from an image-text interleaved format to pure text corpus and image-text pairs. Through comprehensive analysis and experiments, we validate the quality, usability, and effectiveness of the proposed dataset. We hope this could provide a solid data foundation for future multimodal model research. Code and data are released at https://github.com/OpenGVLab/OmniCorpus.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Builds on 102 of its 162 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

102 of 162 references.

References · 102

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2023Qwen Technical Reportcited by 3,273
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  9. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  12. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  13. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  14. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  15. 2019Towards VQA Models That Can Readcited by 1,696
  16. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  17. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  18. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  19. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  20. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  21. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  22. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  23. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  24. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  25. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  26. 2016Modeling Context in Referring Expressionscited by 1,054
  27. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  28. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  29. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  30. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  31. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  32. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  33. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  34. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  35. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  36. 2016A Diagram Is Worth A Dozen Imagescited by 793
  37. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  38. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724

50 most cited of 102 All 102 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 36; 2026: 13.

58 papers.

Sort:

By area:

  1. 2025Qwen3-VL Technical Reportcited by 2,534
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  3. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  4. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  5. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
  6. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
  7. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
  8. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  9. 2025MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipecited by 174
  10. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  11. 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoningcited by 134
  12. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
38 more
  1. 2025Zebra-CoT: A Dataset for Interleaved Vision Language Reasoningcited by 78
  2. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  3. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
  4. 2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 69
  5. 2024Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languagescited by 63
  6. 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Modelscited by 61
  7. 2025LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistantcited by 56
  8. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  9. 2025SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understandingcited by 30
  10. 2025Vlaser: Vision-Language-Action Model with Synergistic Embodied Reasoningcited by 22
  11. 2025Docopilot: Improving Multimodal Models for Document-Level Understandingcited by 19
  12. 2025SRUM: Fine-Grained Self-Rewarding for Unified Multimodal Modelscited by 19
  13. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matchingcited by 19
  14. 2025MMCR: Advancing Visual Language Model in Multimodal Multi-Turn Contextual Reasoningcited by 16
  15. 2025Query-Kontext: An Unified Multimodal Model for Image Generation and Editingcited by 15
  16. 2025Retrieval-augmented GUI Agents with Generative Guidelinescited by 14
  17. 2025Perception in Reflectioncited by 13
  18. 2025GM-PRM: A Generative Multimodal Process Reward Model for Multimodal Mathematical Reasoningcited by 13
  19. 2026NextFlow: Unified Sequential Modeling Activates Multimodal Understanding and Generationcited by 13
  20. 2024Chimera: Improving Generalist Model with Domain-Specific Expertscited by 11
  21. 2025Visuospatial Cognitive Assistantcited by 11
  22. 2026Bernini: Latent Semantic Planning for Video Diffusioncited by 11
  23. 20252.5 Years in Class: A Multimodal Textbook for Vision-Language Pretrainingcited by 9
  24. 2025Comparative Analysis Based on DeepSeek, ChatGPT, and Google Gemini: Features, Techniques, Performance, Future Prospectscited by 9
  25. 2025Reasoning Guided Embeddings: Leveraging MLLM Reasoning for Improved Multimodal Retrievalcited by 9
  26. 2024BLIP3-KALE: Knowledge Augmented Large-Scale Dense Captionscited by 8
  27. 2025RealSyn: An Effective and Scalable Multimodal Interleaved Document Transformation Paradigmcited by 7
  28. 2025LVC: A Lightweight Compression Framework for Enhancing VLMs in Long Video Understandingcited by 6
  29. 2025Growing Visual Generative Capacity for Pre-Trained MLLMscited by 4
  30. 2025AndesVL Technical Report: An Efficient Mobile-side Multimodal Large Language Modelcited by 4
  31. 2025RxnCaption: Reformulating Reaction Diagram Parsing as Visual Prompt Guided Captioningcited by 3
  32. 2025Co-Training Vision Language Models for Remote Sensing Multi-task Learningcited by 3
  33. 2026Look Before Acting: Enhancing Vision Foundation Representations for Vision-Language-Action Modelscited by 3
  34. 2026MACRO: Advancing Multi-Reference Image Generation with Structured Long-Context Datacited by 3
  35. 2024Reflexive Guidance: Improving OoDD in Vision-Language Models via Self-Guided Image-Adaptive Concept Generationcited by 2
  36. 2026DuoGen: Towards General Purpose Interleaved Multimodal Generationcited by 2
  37. 2026CRIT: Graph-Based Automatic Data Synthesis to Enhance Cross-Modal Multi-Hop Reasoningcited by 2
  38. 2026Images in Sentences: Scaling Interleaved Instructions for Unified Visual Generationcited by 2

50 most cited of 58 All 58 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation
  2. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review
  3. 2024MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens
  4. 2025A High-Quality Dataset and Reliable Evaluation for Interleaved Image-Text Generation
  5. 2023CapsFusion: Rethinking Image-Text Data at Scale
  6. 2026MONET: A Massive, Open, Non-redundant and Enriched Text-to-image dataset
  7. 2023Multimodal Large Language Models: A Survey
  8. 2024CoMM: A Coherent Interleaved Image-Text Dataset for Multimodal Understanding and Generation