# FineMath: A Fine-Grained Mathematical Evaluation Benchmark for Chinese Large Language Models

arXiv: https://arxiv.org/abs/2403.07747 · HTML: https://mlsanity.com/paper/2403.07747/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Mathematical reasoning, one of 17 directions in LLM reasoning.
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 12 Mar 2024.

## Placed in

- LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/)

## Builds on (most cited first, 22 of 22)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [Are NLP Models really able to Solve Simple Math Word Problems?](https://mlsanity.com/paper/2103.07191/) (2021)
- [Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problems](https://mlsanity.com/paper/1705.04146/) (2017)
- [Solving General Arithmetic Word Problems](https://mlsanity.com/paper/1608.01413/) (2016)
- [A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers](https://mlsanity.com/paper/2106.15772/) (2021)
- [Large Language Model Alignment: A Survey](https://mlsanity.com/paper/2309.15025/) (2023)
- [Evaluating Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2310.19736/) (2023)
- [CMATH: Can Your Language Model Pass Chinese Elementary School Math Test?](https://mlsanity.com/paper/2306.16636/) (2023)
- [GPT Can Solve Mathematical Problems Without a Calculator](https://mlsanity.com/paper/2309.03241/) (2023)
- [Ape210K: A Large-Scale and Template-Rich Dataset of Math Word Problems](https://mlsanity.com/paper/2009.11506/) (2020)
- [MWP-BERT: Numeracy-Augmented Pre-training for Math Word Problem Solving](https://mlsanity.com/paper/2107.13435/) (2021)
- [MinT: Boosting Generalization in Mathematical Reasoning via Multi-View Fine-Tuning](https://mlsanity.com/paper/2307.07951/) (2023)
- [Recall and Learn: A Memory-augmented Solver for Math Word Problems](https://mlsanity.com/paper/2109.13112/) (2021)
- [Self-consistent Reasoning For Solving Math Word Problems](https://mlsanity.com/paper/2210.15373/) (2022)

## Built on by (most cited first, 10 of 10)

- [MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttraining](https://mlsanity.com/paper/2505.07608/) (2025)
- [MathFlow: Enhancing the Perceptual Flow of MLLMs for Visual Mathematical Problems](https://mlsanity.com/paper/2503.16549/) (2025)
- [Can LLMs Reason Structurally? Benchmarking via the Lens of Data Structures](https://mlsanity.com/paper/2505.24069/) (2025)
- [BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMs](https://mlsanity.com/paper/2604.02045/) (2026)
- [MobileMoE: Scaling On-Device Mixture of Experts](https://mlsanity.com/paper/2605.27358/) (2026)
- [A Perspective on Large Language Models, Intelligent Machines, and Knowledge Acquisition](https://mlsanity.com/paper/2408.06598/) (2024)
- [Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-training](https://mlsanity.com/paper/2602.00747/) (2026)
- [Optimal Sparsity of Mixture-of-Experts Language Models for Reasoning Tasks](https://mlsanity.com/paper/2508.18672/) (2025)
- [FLOP-Efficient Training: Early Stopping Based on Test-Time Compute Awareness](https://mlsanity.com/paper/2601.01332/) (2026)
- [On Meta-Evaluation](https://mlsanity.com/paper/2601.14262/) (2025)

## Abstract (arXiv)

To thoroughly assess the mathematical reasoning abilities of Large Language Models (LLMs), we need to carefully curate evaluation datasets covering diverse mathematical concepts and mathematical problems at different difficulty levels. In pursuit of this objective, we propose FineMath in this paper, a fine-grained mathematical evaluation benchmark dataset for assessing Chinese LLMs. FineMath is created to cover the major key mathematical concepts taught in elementary school math, which are further divided into 17 categories of math word problems, enabling in-depth analysis of mathematical reasoning abilities of LLMs. All the 17 categories of math word problems are manually annotated with their difficulty levels according to the number of reasoning steps required to solve these problems. We conduct extensive experiments on a wide range of LLMs on FineMath and find that there is still considerable room for improvements in terms of mathematical reasoning capability of Chinese LLMs. We also carry out an in-depth analysis on the evaluation process and methods that have been overlooked previously. These two factors significantly influence the model results and our understanding of their mathematical reasoning capabilities. The dataset will be publicly available soon.