# CLUECorpus2020: A Large-scale Chinese Corpus for Pre-training Language Model

arXiv: https://arxiv.org/abs/2003.01355 · HTML: https://mlsanity.com/paper/2003.01355/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Mar 2020.

## Built on by (most cited first, 30 of 36)

- [PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation](https://mlsanity.com/paper/2104.12369/) (2021)
- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [LongAlign: A Recipe for Long Context Alignment of Large Language Models](https://mlsanity.com/paper/2401.18058/) (2024)
- [M6: A Chinese Multimodal Pretrainer](https://mlsanity.com/paper/2103.00823/) (2021)
- [A Survey of Multimodal Large Language Model from A Data-centric Perspective](https://mlsanity.com/paper/2405.16640/) (2024)
- [PanGu-Sigma: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing](https://mlsanity.com/paper/2303.10845/) (2023)
- [Lawyer LLaMA Technical Report](https://mlsanity.com/paper/2305.15062/) (2023)
- [CSL: A Large-scale Chinese Scientific Literature Dataset](https://mlsanity.com/paper/2209.05034/) (2022)
- [SMedBERT: A Knowledge-Enhanced Pre-trained Language Model with Structured Semantics for Medical Text Mining](https://mlsanity.com/paper/2108.08983/) (2021)
- [ZEN 2.0: Continue Training and Adaption for N-gram Enhanced Text Encoders](https://mlsanity.com/paper/2105.01279/) (2021)
- [SeqGPT: An Out-of-the-box Large Language Model for Open Domain Sequence Understanding](https://mlsanity.com/paper/2308.10529/) (2023)
- [Integrating Stock Features and Global Information via Large Language Models for Enhanced Stock Return Prediction](https://mlsanity.com/paper/2310.05627/) (2023)
- [What Can String Probability Tell Us About Grammaticality?](https://mlsanity.com/paper/2510.16227/) (2025)
- [A Slot Is Not Built in One Utterance: Spoken Language Dialogs with Sub-Slots](https://mlsanity.com/paper/2203.10759/) (2022)
- [Benchmarking of DL Libraries and Models on Mobile Devices](https://mlsanity.com/paper/2202.06512/) (2022)
- [A Cognitive Stimulation Dialogue System with Multi-source Knowledge Fusion for Elders with Cognitive Impairment](https://mlsanity.com/paper/2305.08200/) (2023)
- [Disco-Bench: A Discourse-Aware Evaluation Benchmark for Language Modelling](https://mlsanity.com/paper/2307.08074/) (2023)
- [Exploring and Adapting Chinese GPT to Pinyin Input Method](https://mlsanity.com/paper/2203.00249/) (2022)
- [DecorateLM: Data Engineering through Corpus Rating, Tagging, and Editing with Language Models](https://mlsanity.com/paper/2410.05639/) (2024)
- [Automatic Generation of Product-Image Sequence in E-commerce](https://mlsanity.com/paper/2206.12994/) (2022)
- [Counterfactual Recipe Generation: Exploring Compositional Generalization in a Realistic Scenario](https://mlsanity.com/paper/2210.11431/) (2022)
- [CARE-MI: Chinese Benchmark for Misinformation Evaluation in Maternity and Infant Care](https://mlsanity.com/paper/2307.01458/) (2023)
- [Semantic-Enhanced Explainable Finetuning for Open-Domain Dialogues](https://mlsanity.com/paper/2106.03065/) (2021)
- [An Empirical Study on Large Language Models in Accuracy and Robustness under Chinese Industrial Scenarios](https://mlsanity.com/paper/2402.01723/) (2024)
- [Dynamic data sampler for cross-language transfer learning in large language models](https://mlsanity.com/paper/2405.10626/) (2024)
- [PERT: A New Solution to Pinyin to Character Conversion Task](https://mlsanity.com/paper/2205.11737/) (2022)
- [Tracing Privacy Leakage of Language Models to Training Data via Adjusted Influence Functions](https://mlsanity.com/paper/2408.10468/) (2024)
- [CERD: A Comprehensive Chinese Rhetoric Dataset for Rhetorical Understanding and Generation in Essays](https://mlsanity.com/paper/2409.19691/) (2024)
- [Language Models at the Syntax-Semantics Interface: A Case Study of the Long-Distance Binding of Chinese Reflexive ziji](https://mlsanity.com/paper/2504.02116/) (2025)
- [A Survey of Pun Generation: Datasets, Evaluations and Methodologies](https://mlsanity.com/paper/2507.04793/) (2025)

## Abstract (arXiv)

In this paper, we introduce the Chinese corpus from CLUE organization, CLUECorpus2020, a large-scale corpus that can be used directly for self-supervised learning such as pre-training of a language model, or language generation. It has 100G raw corpus with 35 billion Chinese characters, which is retrieved from Common Crawl. To better understand this corpus, we conduct language understanding experiments on both small and large scale, and results show that the models trained on this corpus can achieve excellent performance on Chinese. We release a new Chinese vocabulary with a size of 8K, which is only one-third of the vocabulary size used in Chinese Bert released by Google. It saves computational cost and memory while works as good as original vocabulary. We also release both large and tiny versions of the pre-trained model on this corpus. The former achieves the state-of-the-art result, and the latter retains most precision while accelerating training and prediction speed for eight times compared to Bert-base. To facilitate future work on self-supervised learning on Chinese, we release our dataset, new vocabulary, codes, and pre-trained models on Github.