# The First ChineseBabyLM Challenge: training data-efficient and cognitively plausible language models for Chinese

arXiv: https://arxiv.org/abs/2607.10745 · HTML: https://mlsanity.com/paper/2607.10745/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 28 references that are in the map.
- First posted on arXiv 12 Jul 2026.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [CLUE: A Chinese Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2004.05986/) (2020)
- [Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora](https://mlsanity.com/paper/2504.08165/) (2025)
- [OCNLI: Original Chinese Natural Language Inference](https://mlsanity.com/paper/2010.05444/) (2020)
- [Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehension](https://mlsanity.com/paper/1904.09679/) (2019)
- [Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora](https://mlsanity.com/paper/2412.05149/) (2024)
- [GPT or BERT: why not both?](https://mlsanity.com/paper/2410.24159/) (2024)
- [A Systematic Assessment of Language Models with Linguistic Minimal Pairs in Chinese](https://mlsanity.com/paper/2411.06096/) (2024)
- [XCOMPS: A Multilingual Benchmark of Conceptual Minimal Pairs](https://mlsanity.com/paper/2502.19737/) (2025)
- [CCI3.0-HQ: a large-scale Chinese dataset of high quality designed for pre-training large language models](https://mlsanity.com/paper/2410.18505/) (2024)
- [The Impact of Visual Information in Chinese Characters: Evaluating Large Models' Ability to Recognize and Utilize Radicals](https://mlsanity.com/paper/2410.09013/) (2024)
- [BabyBabelLM: A Multilingual Benchmark of Developmentally Plausible Training Data](https://mlsanity.com/paper/2510.10159/) (2025)
- [MulCogBench: A Multi-modal Cognitive Benchmark Dataset for Evaluating Chinese and English Computational Language Models](https://mlsanity.com/paper/2403.01116/) (2024)
- [Investigating Transfer Learning in Multilingual Pre-trained Language Models through Chinese Natural Language Inference](https://mlsanity.com/paper/2106.03983/) (2021)
- [Navigating Brain Language Representations: A Comparative Analysis of Neural Language Models and Psychologically Plausible Models](https://mlsanity.com/paper/2404.19364/) (2024)

## Abstract (arXiv)

This paper presents the first ChineseBabyLM Challenge, organized as part of NLPCC 2026. The challenge asked participants to train language models from scratch using no more than 102M Chinese words. The models were evaluated on three tracks: natural language understanding, cognitive alignment, and Hanzi knowledge. There were no restrictions on tokenizers, model architectures, or the number of training epochs. Eighteen teams submitted 28 distinct models, generating 74 result files. The overall-winning team used a DeBERTa-v2 architecture and introduced an auxiliary pinyin-prediction objective during pretraining. Several submissions also explored curriculum-learning strategies and architectural innovations. Overall, the challenge provides a benchmark for advancing data-efficient and cognitively plausible approaches to Chinese language modeling.