mlsanityData to 7 Oct 2026

Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Text

arXiv 2304.06939 ↗First posted 14 Apr 2023cs.CV cs.CLcited by 208

Not on the map. Why?

Cited by 113 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In-context vision and language models like Flamingo support arbitrarily interleaved sequences of images and text as input. This format not only enables few-shot learning via interleaving independent supervised (image, text) examples, but also, more complex prompts involving interaction between images, e.g., "What do image A and image B have in common?" To support this interface, pretraining occurs over web corpora that similarly contain interleaved images+text. To date, however, large-scale data of this form have not been publicly available. We release Multimodal C4, an augmentation of the popular text-only C4 corpus with images interleaved. We use a linear assignment algorithm to place images into longer bodies of text using CLIP features, a process that we show outperforms alternatives. Multimodal C4 spans everyday topics like cooking, travel, technology, etc. A manual inspection of a random sample of documents shows that a vast majority (88%) of images are topically relevant, and that linear assignment frequently selects individual sentences specifically well-aligned with each image (80%). After filtering NSFW images, ads, etc., the resulting corpus consists of 101.2M documents with 571M images interleaved in 43B English tokens.

Key facts. Cited by 208 papers in the map (data through 7 Oct 2026). Builds on 19 of its 34 references that are in the map. First posted on arXiv 14 Apr 2023.

Builds on

19 of 34 references.

References · 19

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  6. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  7. 2018Datasheets for Datasetscited by 1,371
  8. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  9. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  10. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  11. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 326
  12. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
7 more
  1. 2015YFCC100M: The New Data in Multimedia Researchcited by 234
  2. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 175
  3. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 162
  4. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  5. 2019Unsupervised Discovery of Multimodal Links in Multi-image, Multi-sentence Documentscited by 9
  6. 2023Users are the North Star for AI Transparencycited by 5
  7. 2021An Unsupervised Sampling Approach for Image-Sentence Matching Using Document-Level Structural Informationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 48; 2024: 89; 2025: 53; 2026: 18.

208 papers.

Sort:

By area:

  1. 2025Qwen3-VL Technical Reportcited by 2,534
  2. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  3. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  4. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  5. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  6. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  7. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  8. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  9. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  10. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  11. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  12. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
38 more
  1. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  2. 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
  3. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  4. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
  5. 2025Kimi-VL Technical Reportcited by 381
  6. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  7. 2024What matters when building vision-language models?cited by 321
  8. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  9. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  10. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 297
  11. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  12. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  13. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  14. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  15. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  16. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
  17. 2025Transfer between Modalities with MetaQueriescited by 231
  18. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 213
  19. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 212
  20. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  21. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 201
  22. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 184
  23. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 178
  24. 2023Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Datacited by 176
  25. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  26. 2023SVIT: Scaling up Visual Instruction Tuningcited by 161
  27. 2023Emu: Generative Pretraining in Multimodalitycited by 156
  28. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  29. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  30. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  31. 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoningcited by 134
  32. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  33. 2024Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Datasetcited by 126
  34. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 125
  35. 2023Dolphins: Multimodal Language Model for Drivingcited by 124
  36. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 124
  37. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  38. 2025StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modelingcited by 107

50 most cited of 208 All 208 (JSON)