mlsanityData to 7 Oct 2026

OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents

arXiv 2306.16527 ↗First posted 21 Jun 2023cs.IR cs.CVcited by 316

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large multimodal models trained on natural documents, which interleave images and text, outperform models trained on image-text pairs on various multimodal benchmarks. However, the datasets used to train these models have not been released, and the collection process has not been fully specified. We introduce the OBELICS dataset, an open web-scale filtered dataset of interleaved image-text documents comprising 141 million web pages extracted from Common Crawl, 353 million associated images, and 115 billion text tokens. We describe the dataset creation process, present comprehensive filtering rules, and provide an analysis of the dataset's content. To show the viability of OBELICS, we train vision and language models of 9 and 80 billion parameters named IDEFICS, and obtain competitive performance on different multimodal benchmarks. We release our dataset, models and code.

Key facts. Cited by 316 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 72 of its 211 references that are in the map. First posted on arXiv 21 Jun 2023.

Builds on

72 of 211 references.

References · 72

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2020Scaling Laws for Neural Language Modelscited by 6,411
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  9. 2015VQA: Visual Question Answeringcited by 2,489
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  11. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  12. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  13. 2017Decoupled Weight Decay Regularizationcited by 1,722
  14. 2019Towards VQA Models That Can Readcited by 1,696
  15. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  16. 2019Root Mean Square Layer Normalizationcited by 1,425
  17. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  18. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  19. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  20. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  21. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  22. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  23. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  24. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  25. 2023StarCoder: may the source be with you!cited by 790
  26. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  27. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  28. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  29. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  30. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  31. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  32. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  33. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  34. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  35. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  36. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  37. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  38. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 33; 2024: 150; 2025: 97; 2026: 36.

316 papers.

Sort:

By area:

  1. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  2. 2025Qwen3-VL Technical Reportcited by 2,534
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  4. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  5. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  6. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  7. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  8. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  9. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  10. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  11. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  12. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
38 more
  1. 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
  2. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  3. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
  4. 2025Kimi-VL Technical Reportcited by 381
  5. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  6. 2024What matters when building vision-language models?cited by 321
  7. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  8. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  9. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 284
  10. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 255
  11. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  12. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
  13. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 233
  14. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  15. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
  16. 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 190
  17. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
  18. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 184
  19. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 178
  20. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 172
  21. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  22. 2024Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Modelscited by 151
  23. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehensioncited by 144
  24. 2025Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Bettercited by 142
  25. 2024Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Modelscited by 138
  26. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  27. 2024The Well: a Large-Scale Collection of Diverse Physics Simulations for Machine Learningcited by 134
  28. 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoningcited by 134
  29. 2024LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Modelcited by 128
  30. 2024Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Datasetcited by 126
  31. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 124
  32. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 120
  33. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
  34. 2025ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Modelcited by 112
  35. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 107
  36. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  37. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Usecited by 100
  38. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Modelcited by 100

50 most cited of 316 All 316 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026MONET: A Massive, Open, Non-redundant and Enriched Text-to-image dataset
  2. 2025Document Haystack: A Long Context Multimodal Image/Document Understanding Vision LLM Benchmark
  3. 2024Finetuned Multimodal Language Models Are High-Quality Image-Text Data Filters
  4. 2024MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens
  5. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions
  6. 2025Wasm: A Pipeline for Constructing Structured Arabic Interleaved Multimodal Corpora
  7. 2025Towards Text-Image Interleaved Retrieval
  8. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review