PCMind-2.1-Kaiyuan-2B Technical Report
Not on the map. Why?
Abstract (arXiv)
The rapid advancement of Large Language Models (LLMs) has resulted in a significant knowledge gap between the open-source community and industry, primarily because the latter relies on closed-source, high-quality data and training recipes. To address this, we introduce PCMind-2.1-Kaiyuan-2B, a fully open-source 2-billion-parameter model focused on improving training efficiency and effectiveness under resource constraints. Our methodology includes three key innovations: a Quantile Data Benchmarking method for systematically comparing heterogeneous open-source datasets and providing insights on data mixing strategies; a Strategic Selective Repetition scheme within a multi-phase paradigm to effectively leverage sparse, high-quality data; and a Multi-Domain Curriculum Training policy that orders samples by quality. Supported by a highly optimized data preprocessing pipeline and architectural modifications for FP16 stability, Kaiyuan-2B achieves performance competitive with state-of-the-art fully open-source models, demonstrating practical and scalable solutions for resource-limited pretraining. We release all assets (including model weights, data, and code) under Apache 2.0 license at https://huggingface.co/thu-pacman/PCMind-2.1-Kaiyuan-2B.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 53 of its 78 references that are in the map. First posted on arXiv 8 Dec 2025.
Builds on
53 of 78 references.
References · 53
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
- 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
- 2021Program Synthesis with Large Language Modelscited by 3,637
- 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
- 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
- 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
38 more
- 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
- 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
- 2019Root Mean Square Layer Normalizationcited by 1,480
- 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
- 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
- 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
- 2020Aligning AI With Shared Human Valuescited by 723
- 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
- 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
- 2016Language Modeling with Gated Convolutional Networkscited by 702
- 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
- 2024OLMo: Accelerating the Science of Language Modelscited by 613
- 2016Neural Combinatorial Optimization with Reinforcement Learningcited by 608
- 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
- 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
- 2023Scaling Data-Constrained Language Modelscited by 403
- 2023Llemma: An Open Language Model For Mathematicscited by 388
- 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
- 2022The Stack: 3 TB of permissively licensed source codecited by 359
- 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
- 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
- 2020Query-Key Normalization for Transformerscited by 280
- 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 170
- 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 160
- 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Textcited by 155
- 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 150
- 2023Skywork: A More Open Bilingual Foundation Modelcited by 112
- 2024OLMES: A Standard for Language Model Evaluationscited by 106
- 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 104
- 2025Organize the Web: Constructing Domains Enhances Pre-Training Data Curationcited by 76
- 2020CPM: A Large-scale Generative Chinese Pre-trained Language Modelcited by 60
- 2021CPM-2: Large-scale Cost-effective Pre-trained Language Modelscited by 52
- 2023WanJuan: A Comprehensive Multimodal Dataset for Advancing English and Chinese Large Modelscited by 50
- 2024OpenDataLab: Empowering General Artificial Intelligence with Open Datasetscited by 29
- 2024Autonomous Data Selection with Zero-shot Generative Classifiers for Mathematical Textscited by 27
- 2025WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-trainingcited by 23
- 2025OpenCSG Chinese Corpus: A Series of High-quality Chinese Datasets for LLM Trainingcited by 21
- 2025Rewriting Pre-Training Data Boosts LLM Performance in Math and Codecited by 21
50 most cited of 53 All 53 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.