mlsanityData to 7 Oct 2026

WanJuan-CC: A Safe and High-Quality Open-sourced English Webtext Dataset

arXiv 2402.19282 ↗First posted 29 Feb 2024cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

This paper presents WanJuan-CC, a safe and high-quality open-sourced English webtext dataset derived from Common Crawl data. The study addresses the challenges of constructing large-scale pre-training datasets for language models, which require vast amounts of high-quality data. A comprehensive process was designed to handle Common Crawl data, including extraction, heuristic rule filtering, fuzzy deduplication, content safety filtering, and data quality filtering. From approximately 68 billion original English documents, we obtained 2.22T Tokens of safe data and selected 1.0T Tokens of high-quality data as part of WanJuan-CC. We have open-sourced 100B Tokens from this dataset. The paper also provides statistical information related to data quality, enabling users to select appropriate data according to their needs. To evaluate the quality and utility of the dataset, we trained 1B-parameter and 3B-parameter models using WanJuan-CC and another dataset, RefinedWeb. Results show that WanJuan-CC performs better on validation datasets and downstream tasks.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 14 of its 30 references that are in the map. First posted on arXiv 29 Feb 2024.

Builds on

14 of 30 references.

References · 14

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  7. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  8. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  9. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  10. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  11. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  12. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
2 more
  1. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  2. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 146

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 5; 2026: 2.

10 papers.

  1. 2024OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Textcited by 58
  2. 2024Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scalecited by 37
  3. 2025Ultra-FineWeb: Efficient Data Filtering and Verification for High-Quality LLM Training Datacited by 35
  4. 2025Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Modelscited by 19
  5. 2025RedOne: Revealing Domain-specific LLM Post-Training in Social Networking Servicescited by 6
  6. 2024Dial-insight: Fine-tuning Large Language Models with High-Quality Domain-Specific Data Preventing Capability Collapsecited by 4
  7. 2025Innovator: Scientific Continued Pretraining with Fine-grained MoE Upcyclingcited by 4
  8. 2025Alchemist: Unlocking Efficiency in Text-to-Image Model Training via Meta-Gradient Data Selectioncited by 1
  9. 2026Explicit Multi-head Attention for Inter-head Interaction in Large Language Modelscited by 1
  10. 2026UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editingcited by 1