# JiuZhang: A Chinese Pre-trained Language Model for Mathematical Problem Understanding

arXiv: https://arxiv.org/abs/2206.06315 · HTML: https://mlsanity.com/paper/2206.06315/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 38 references that are in the map.
- First posted on arXiv 13 Jun 2022.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [Generative Language Modeling for Automated Theorem Proving](https://mlsanity.com/paper/2009.03393/) (2020)
- [ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation](https://mlsanity.com/paper/2107.02137/) (2021)
- [Formal Mathematics Statement Curriculum Learning](https://mlsanity.com/paper/2202.01344/) (2022)
- [PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation](https://mlsanity.com/paper/2104.12369/) (2021)
- [CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation](https://mlsanity.com/paper/2109.05729/) (2021)
- [ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information](https://mlsanity.com/paper/2106.16038/) (2021)
- [CPM: A Large-scale Generative Chinese Pre-trained Language Model](https://mlsanity.com/paper/2012.00413/) (2020)
- [MathBERT: A Pre-Trained Model for Mathematical Formula Understanding](https://mlsanity.com/paper/2105.00377/) (2021)
- [Semantically-Aligned Equation Generation for Solving and Reasoning Math Word Problems](https://mlsanity.com/paper/1811.00720/) (2018)
- [Mengzi: Towards Lightweight yet Ingenious Pre-trained Models for Chinese](https://mlsanity.com/paper/2110.06696/) (2021)
- [MWPToolkit: An Open-Source Framework for Deep Learning-Based Math Word Problem Solvers](https://mlsanity.com/paper/2109.00799/) (2021)
- [MathBERT: A Pre-trained Language Model for General NLP Tasks in Mathematics Education](https://mlsanity.com/paper/2106.07340/) (2021)

## Built on by (most cited first, 15 of 15)

- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Future](https://mlsanity.com/paper/2309.15402/) (2023)
- [A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challenges](https://mlsanity.com/paper/2412.11936/) (2024)
- [Evaluating and Improving Tool-Augmented Computation-Intensive Math Reasoning](https://mlsanity.com/paper/2306.02408/) (2023)
- [JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Models](https://mlsanity.com/paper/2405.14365/) (2024)
- [Improving Large Language Models via Fine-grained Reinforcement Learning with Minimum Editing Constraint](https://mlsanity.com/paper/2401.06081/) (2024)
- [Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoning](https://mlsanity.com/paper/2502.02871/) (2025)
- [Towards a Holistic Understanding of Mathematical Questions with Contrastive Pre-training](https://mlsanity.com/paper/2301.07558/) (2023)
- [Leveraging Label Semantics and Meta-Label Refinement for Multi-Label Question Classification](https://mlsanity.com/paper/2411.01841/) (2024)
- [Modeling Complex Mathematical Reasoning via Large Language Model based MathAgent](https://mlsanity.com/paper/2312.08926/) (2023)
- [JiuZhang 2.0: A Unified Chinese Pre-trained Language Model for Multi-task Mathematical Problem Solving](https://mlsanity.com/paper/2306.11027/) (2023)
- [VLLaVO: Mitigating Visual Gap through LLMs](https://mlsanity.com/paper/2401.03253/) (2024)
- [EduNLP: Towards a Unified and Modularized Library for Educational Resources](https://mlsanity.com/paper/2406.01276/) (2024)
- [A Knowledge-Injected Curriculum Pretraining Framework for Question Answering](https://mlsanity.com/paper/2403.09712/) (2024)
- [Iterative In-Context Learning to Enhance LLMs Abstract Reasoning: The Case-Study of Algebraic Tasks](https://mlsanity.com/paper/2509.01267/) (2025)

## Abstract (arXiv)

This paper aims to advance the mathematical intelligence of machines by presenting the first Chinese mathematical pre-trained language model~(PLM) for effectively understanding and representing mathematical problems. Unlike other standard NLP tasks, mathematical texts are difficult to understand, since they involve mathematical terminology, symbols and formulas in the problem statement. Typically, it requires complex mathematical logic and background knowledge for solving mathematical problems. Considering the complex nature of mathematical texts, we design a novel curriculum pre-training approach for improving the learning of mathematical PLMs, consisting of both basic and advanced courses. Specially, we first perform token-level pre-training based on a position-biased masking strategy, and then design logic-based pre-training tasks that aim to recover the shuffled sentences and formulas, respectively. Finally, we introduce a more difficult pre-training task that enforces the PLM to detect and correct the errors in its generated solutions. We conduct extensive experiments on offline evaluation (including nine math-related tasks) and online A/B test. Experimental results demonstrate the effectiveness of our approach compared with a number of competitive baselines. Our code is available at: textcolorblueurlhttps://github.com/RUCAIBox/JiuZhang.