# WanJuan: A Comprehensive Multimodal Dataset for Advancing English and Chinese Large Models

arXiv: https://arxiv.org/abs/2308.10755 · HTML: https://mlsanity.com/paper/2308.10755/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 12 of its 18 references that are in the map.
- First posted on arXiv 21 Aug 2023.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Text](https://mlsanity.com/paper/2304.06939/) (2023)

## Built on by (most cited first, 30 of 50)

- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding](https://mlsanity.com/paper/2412.10302/) (2024)
- [InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model](https://mlsanity.com/paper/2401.16420/) (2024)
- [InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition](https://mlsanity.com/paper/2309.15112/) (2023)
- [InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output](https://mlsanity.com/paper/2407.03320/) (2024)
- [InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD](https://mlsanity.com/paper/2404.06512/) (2024)
- [Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training](https://mlsanity.com/paper/2410.08202/) (2024)
- [VIGC: Visual Instruction Generation and Correction](https://mlsanity.com/paper/2308.12714/) (2023)
- [Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers](https://mlsanity.com/paper/2404.04925/) (2024)
- [Memory3: Language Modeling with Explicit Memory](https://mlsanity.com/paper/2407.01178/) (2024)
- [MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Models](https://mlsanity.com/paper/2406.07594/) (2024)
- [TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokens](https://mlsanity.com/paper/2410.05261/) (2024)
- [OpenDataLab: Empowering General Artificial Intelligence with Open Datasets](https://mlsanity.com/paper/2407.13773/) (2024)
- [EVLM: An Efficient Vision-Language Model for Visual Understanding](https://mlsanity.com/paper/2407.14177/) (2024)
- [The Evolution of Multimodal Model Architectures](https://mlsanity.com/paper/2405.17927/) (2024)
- [Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Models](https://mlsanity.com/paper/2504.14194/) (2025)
- [From Pixels to Words -- Towards Native Vision-Language Primitives at Scale](https://mlsanity.com/paper/2510.14979/) (2025)
- [Ziya2: Data-centric Learning is All LLMs Need](https://mlsanity.com/paper/2311.03301/) (2023)
- [DualFocus: Integrating Macro and Micro Perspectives in Multi-modal Large Language Models](https://mlsanity.com/paper/2402.14767/) (2024)
- [TextPecker: Rewarding Structural Anomaly Quantification for Enhancing Visual Text Rendering](https://mlsanity.com/paper/2602.20903/) (2026)
- [Mono-InternVL-1.5: Towards Cheaper and Faster Monolithic Multimodal Large Language Models](https://mlsanity.com/paper/2507.12566/) (2025)
- [MindLLM: Pre-training Lightweight Large Language Model from Scratch, Evaluations and Domain Applications](https://mlsanity.com/paper/2310.15777/) (2023)
- [LongWanjuan: Towards Systematic Measurement for Long Text Quality](https://mlsanity.com/paper/2402.13583/) (2024)
- [SampleMix: A Sample-wise Pre-training Data Mixing Strategey by Coordinating Data Quality and Diversity](https://mlsanity.com/paper/2503.01506/) (2025)
- [YUAN 2.0: A Large Language Model with Localized Filtering-based Attention](https://mlsanity.com/paper/2311.15786/) (2023)
- [SentenceVAE: Enable Next-sentence Prediction for Large Language Models with Faster Speed, Higher Accuracy and Longer Context](https://mlsanity.com/paper/2408.00655/) (2024)
- [YuLan-Mini: An Open Data-efficient Language Model](https://mlsanity.com/paper/2412.17743/) (2024)
- [VisioMath: Benchmarking Figure-based Mathematical Reasoning in LMMs](https://mlsanity.com/paper/2506.06727/) (2025)
- [MiLiC-Eval: Benchmarking Multilingual LLMs for China's Minority Languages](https://mlsanity.com/paper/2503.01150/) (2025)

## Abstract (arXiv)

The rise in popularity of ChatGPT and GPT-4 has significantly accelerated the development of large models, leading to the creation of numerous impressive large language models(LLMs) and multimodal large language models (MLLMs). These cutting-edge models owe their remarkable performance to high-quality data. However, the details of the training data used in leading paradigms are often kept confidential. This lack of transparency, coupled with the scarcity of open-source data, impedes further developments within the community. As a response, this paper presents "Wan Juan", a large-scale multimodal dataset composed of both Chinese and English data, collected from a wide range of web sources. The dataset incorporates text, image-text, and video modalities, with a total volume exceeding 2TB. It was utilized in the training of InternLM, a model that demonstrated significant advantages in multi-dimensional evaluations when compared to models of a similar scale. All data can be accessed at https://opendatalab.org.cn/WanJuan1.0.