mlsanityData to 14 Oct 2026

CCI3.0-HQ: a large-scale Chinese dataset of high quality designed for pre-training large language models

arXiv 2410.18505 ↗First posted 24 Oct 2024cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

We present CCI3.0-HQ (https://huggingface.co/datasets/BAAI/CCI3-HQ), a high-quality 500GB subset of the Chinese Corpora Internet 3.0 (CCI3.0)(https://huggingface.co/datasets/BAAI/CCI3-Data), developed using a novel two-stage hybrid filtering pipeline that significantly enhances data quality. To evaluate its effectiveness, we trained a 0.5B parameter model from scratch on 100B tokens across various datasets, achieving superior performance on 10 benchmarks in a zero-shot setting compared to CCI3.0, SkyPile, and WanjuanV1. The high-quality filtering process effectively distills the capabilities of the Qwen2-72B-instruct model into a compact 0.5B model, attaining optimal F1 scores for Chinese web data classification. We believe this open-access dataset will facilitate broader access to high-quality language models.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 8 of its 20 references that are in the map. First posted on arXiv 24 Oct 2024.

Builds on

8 of 20 references.

References · 8

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  4. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  6. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  7. 2023Skywork: A More Open Bilingual Foundation Modelcited by 112
  8. 2019Careful Selection of Knowledge to solve Open Book Question Answeringcited by 35

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 6.

11 papers.

  1. 2025MiniCPM4: Ultra-Efficient LLMs on End Devicescited by 52
  2. 2025Ultra-FineWeb: Efficient Data Filtering and Verification for High-Quality LLM Training Datacited by 38
  3. 2026WebWorld: A Large-Scale World Model for Web Agent Trainingcited by 26
  4. 2025Uni-X: Mitigating Modality Conflict with a Two-End-Separated Architecture for Unified Multimodal Modelscited by 18
  5. 2025Citrus: Leveraging Expert Cognitive Pathways in a Medical Language Model for Advanced Medical Decision Supportcited by 16
  6. 2026Data Science and Technology Towards AGI Part I: Tiered Data Managementcited by 4
  7. 2026Towards Privacy-Preserving LLM Inference via Covariant Obfuscation (Technical Report)cited by 2
  8. 2025MoM: Mixtures of Scenario-Aware Document Memories for Retrieval-Augmented Generation Systemscited by 1
  9. 2026UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editingcited by 1
  10. 2026Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within 5090cited by 1
  11. 2026The First ChineseBabyLM Challenge: training data-efficient and cognitively plausible language models for Chinesecited by 0