# CSL: A Large-scale Chinese Scientific Literature Dataset

arXiv: https://arxiv.org/abs/2209.05034 · HTML: https://mlsanity.com/paper/2209.05034/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 29 references that are in the map.
- First posted on arXiv 12 Sep 2022.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP](https://mlsanity.com/paper/2104.08835/) (2021)
- [Deep Keyphrase Generation](https://mlsanity.com/paper/1704.06879/) (2017)
- [FLEX: Unifying Evaluation for Few-Shot NLP](https://mlsanity.com/paper/2107.07170/) (2021)
- [LCSTS: A Large Scale Chinese Short Text Summarization Dataset](https://mlsanity.com/paper/1506.05865/) (2015)
- [UER: An Open-Source Toolkit for Pre-training Models](https://mlsanity.com/paper/1909.05658/) (2019)
- [CLUECorpus2020: A Large-scale Chinese Corpus for Pre-training Language Model](https://mlsanity.com/paper/2003.01355/) (2020)
- [Scientific document summarization via citation contextualization and scientific discourse](https://mlsanity.com/paper/1706.03449/) (2017)

## Built on by (most cited first, 30 of 45)

- [Making Text Embedders Few-Shot Learners](https://mlsanity.com/paper/2409.15700/) (2024)
- [ChatGLM-Math: Improving Math Problem-Solving in Large Language Models with a Self-Critique Pipeline](https://mlsanity.com/paper/2404.02893/) (2024)
- [Origin Tracing and Detecting of LLMs](https://mlsanity.com/paper/2304.14072/) (2023)
- [KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Model](https://mlsanity.com/paper/2506.20923/) (2025)
- [Chinese Open Instruction Generalist: A Preliminary Release](https://mlsanity.com/paper/2304.07987/) (2023)
- [The ACL OCL Corpus: Advancing Open Science in Computational Linguistics](https://mlsanity.com/paper/2305.14996/) (2023)
- [Integrating Stock Features and Global Information via Large Language Models for Enhanced Stock Return Prediction](https://mlsanity.com/paper/2310.05627/) (2023)
- [PanGu-π: Enhancing Language Model Architectures via Nonlinearity Compensation](https://mlsanity.com/paper/2312.17276/) (2023)
- [EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models](https://mlsanity.com/paper/2409.17892/) (2024)
- [Is ChatGPT Involved in Texts? Measure the Polish Ratio to Detect ChatGPT-Generated Text](https://mlsanity.com/paper/2307.11380/) (2023)
- [TeleChat Technical Report](https://mlsanity.com/paper/2401.03804/) (2024)
- [Evaluating Large Language Models for Radiology Natural Language Processing](https://mlsanity.com/paper/2307.13693/) (2023)
- [Tele-FLM Technical Report](https://mlsanity.com/paper/2404.16645/) (2024)
- [More is not always better? Enhancing Many-Shot In-Context Learning with Differentiated and Reweighting Objectives](https://mlsanity.com/paper/2501.04070/) (2025)
- [F2LLM-v2: Inclusive, Performant, and Efficient Embeddings for a Multilingual World](https://mlsanity.com/paper/2603.19223/) (2026)
- [MindLLM: Pre-training Lightweight Large Language Model from Scratch, Evaluations and Domain Applications](https://mlsanity.com/paper/2310.15777/) (2023)
- [Piccolo2: General Text Embedding with Multi-task Hybrid Loss Training](https://mlsanity.com/paper/2405.06932/) (2024)
- [NaSGEC: a Multi-Domain Chinese Grammatical Error Correction Dataset from Native Speaker Texts](https://mlsanity.com/paper/2305.16023/) (2023)
- [AutoMIR: Effective Zero-Shot Medical Information Retrieval without Relevance Labels](https://mlsanity.com/paper/2410.20050/) (2024)
- [OpenBA: An Open-sourced 15B Bilingual Asymmetric seq2seq Model Pre-trained from Scratch](https://mlsanity.com/paper/2309.10706/) (2023)
- [A Survey of Pre-trained Language Models for Processing Scientific Text](https://mlsanity.com/paper/2401.17824/) (2024)
- [Soulstyler: Using Large Language Model to Guide Image Style Transfer for Target Object](https://mlsanity.com/paper/2311.13562/) (2023)
- [CARE-MI: Chinese Benchmark for Misinformation Evaluation in Maternity and Infant Care](https://mlsanity.com/paper/2307.01458/) (2023)
- [InternLM-Law: An Open Source Chinese Legal Large Language Model](https://mlsanity.com/paper/2406.14887/) (2024)
- [ZhuJiu: A Multi-dimensional, Multi-faceted Chinese Benchmark for Large Language Models](https://mlsanity.com/paper/2308.14353/) (2023)
- [Continuous Training and Fine-tuning for Domain-Specific Language Models in Medical Question Answering](https://mlsanity.com/paper/2311.00204/) (2023)
- [Head-wise Shareable Attention for Large Language Models](https://mlsanity.com/paper/2402.11819/) (2024)
- [Massively Multilingual Adaptation of Large Language Models Using Bilingual Translation Data](https://mlsanity.com/paper/2506.00469/) (2025)
- [ERNIE 3.0 Tiny: Frustratingly Simple Method to Improve Task-Agnostic Distillation Generalization](https://mlsanity.com/paper/2301.03416/) (2023)
- [An Empirical Study on Large Language Models in Accuracy and Robustness under Chinese Industrial Scenarios](https://mlsanity.com/paper/2402.01723/) (2024)

## Abstract (arXiv)

Scientific literature serves as a high-quality corpus, supporting a lot of Natural Language Processing (NLP) research. However, existing datasets are centered around the English language, which restricts the development of Chinese scientific NLP. In this work, we present CSL, a large-scale Chinese Scientific Literature dataset, which contains the titles, abstracts, keywords and academic fields of 396k papers. To our knowledge, CSL is the first scientific document dataset in Chinese. The CSL can serve as a Chinese corpus. Also, this semi-structured data is a natural annotation that can constitute many supervised NLP tasks. Based on CSL, we present a benchmark to evaluate the performance of models across scientific domain tasks, i.e., summarization, keyword generation and text classification. We analyze the behavior of existing text-to-text models on the evaluation tasks and reveal the challenges for Chinese scientific NLP tasks, which provides a valuable reference for future research. Data and code are available at https://github.com/ydli-ai/CSL