# OpenCSG Chinese Corpus: A Series of High-quality Chinese Datasets for LLM Training

arXiv: https://arxiv.org/abs/2501.08197 · HTML: https://mlsanity.com/paper/2501.08197/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jan 2025.

## Built on by (most cited first, 21 of 21)

- [Dynamic Chunking for End-to-End Hierarchical Sequence Modeling](https://mlsanity.com/paper/2507.07955/) (2025)
- [MiniCPM4: Ultra-Efficient LLMs on End Devices](https://mlsanity.com/paper/2506.07900/) (2025)
- [Ultra-FineWeb: Efficient Data Filtering and Verification for High-Quality LLM Training Data](https://mlsanity.com/paper/2505.05427/) (2025)
- [Investigating and Scaling up Code-Switching for Multilingual Language Model Pre-Training](https://mlsanity.com/paper/2504.01801/) (2025)
- [How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining](https://mlsanity.com/paper/2511.18903/) (2025)
- [FDLLM: A Dedicated Detector for Black-Box LLMs Fingerprinting](https://mlsanity.com/paper/2501.16029/) (2025)
- [Data Science and Technology Towards AGI Part I: Tiered Data Management](https://mlsanity.com/paper/2602.09003/) (2026)
- [Expert Divergence Learning for MoE-based Language Models](https://mlsanity.com/paper/2603.00054/) (2026)
- [Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling](https://mlsanity.com/paper/2604.25578/) (2026)
- [Assessing the Role of Data Quality in Training Bilingual Language Models](https://mlsanity.com/paper/2506.12966/) (2025)
- [PCMind-2.1-Kaiyuan-2B Technical Report](https://mlsanity.com/paper/2512.07612/) (2025)
- [CAPC-CG: A Large-Scale, Expert-Directed LLM-Annotated Corpus of Adaptive Policy Communication in China](https://mlsanity.com/paper/2510.08986/) (2025)
- [UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing](https://mlsanity.com/paper/2607.08646/) (2026)
- [Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within 5090](https://mlsanity.com/paper/2608.27370/) (2026)
- [Chinese ModernBERT with Whole-Word Masking](https://mlsanity.com/paper/2510.12285/) (2025)
- [Peek2: Regex-free Byte-level Byte-Pair Encoding Pretokenizer for LLM Inference on Edge Devices](https://mlsanity.com/paper/2601.05833/) (2026)
- [Pretraining A Large Language Model using Distributed GPUs: A Memory-Efficient Decentralized Paradigm](https://mlsanity.com/paper/2602.11543/) (2026)
- [Mi:dm K 2.5 Pro](https://mlsanity.com/paper/2603.18788/) (2026)
- [Speculative Decoding and the Curse of Multilinguality](https://mlsanity.com/paper/2605.30580/) (2026)
- [ConSA: Controllable Sparsity in Hybrid Attention via Learnable Allocation](https://mlsanity.com/paper/2606.18056/) (2026)
- [The Price of Token Boundaries: Compression Certificates and Prediction](https://mlsanity.com/paper/2609.35869/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable capabilities, but their success heavily relies on the quality of pretraining corpora. For Chinese LLMs, the scarcity of high-quality Chinese datasets presents a significant challenge, often limiting their performance. To address this issue, we propose the OpenCSG Chinese Corpus, a series of high-quality datasets specifically designed for LLM pretraining, post-training, and fine-tuning. This corpus includes Fineweb-edu-chinese, Fineweb-edu-chinese-v2, Cosmopedia-chinese, and Smoltalk-chinese, each with distinct characteristics: Fineweb-edu datasets focus on filtered, high-quality content derived from diverse Chinese web sources; Cosmopedia-chinese provides synthetic, textbook-style data for knowledge-intensive training; and Smoltalk-chinese emphasizes stylistic and diverse chat-format data. The OpenCSG Chinese Corpus is characterized by its high-quality text, diverse coverage across domains, and scalable, reproducible data curation processes. Additionally, we conducted extensive experimental analyses, including evaluations on smaller parameter models, which demonstrated significant performance improvements in tasks such as C-Eval, showcasing the effectiveness of the corpus for training Chinese LLMs.