mlsanityData to 7 Oct 2026

CMATH: Can Your Language Model Pass Chinese Elementary School Math Test?

arXiv 2306.16636 ↗First posted 29 Jun 2023cs.CL cs.AI cs.LGcited by 89

Not on the map. Why?

Cited by 34 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present the Chinese Elementary School Math Word Problems (CMATH) dataset, comprising 1.7k elementary school-level math word problems with detailed annotations, source from actual Chinese workbooks and exams. This dataset aims to provide a benchmark tool for assessing the following question: to what grade level of elementary school math do the abilities of popular large language models (LLMs) correspond? We evaluate a variety of popular LLMs, including both commercial and open-source options, and discover that only GPT-4 achieves success (accuracy ≥ 60%) across all six elementary school grades, while other models falter at different grade levels. Furthermore, we assess the robustness of several top-performing LLMs by augmenting the original problems in the CMATH dataset with distracting information. Our findings reveal that GPT-4 is able to maintains robustness, while other model fail. We anticipate that our study will expose limitations in LLMs' arithmetic and reasoning capabilities, and promote their ongoing development and advancement.

Key facts. Cited by 89 papers in the map (data through 7 Oct 2026). Builds on 16 of its 24 references that are in the map. First posted on arXiv 29 Jun 2023.

Builds on

16 of 24 references.

References · 16

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  7. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  8. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  9. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  10. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  11. 2022Galactica: A Large Language Model for Sciencecited by 736
  12. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
4 more
  1. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  2. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  3. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 302
  4. 2020Ape210K: A Large-Scale and Template-Rich Dataset of Math Word Problemscited by 60

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 21; 2025: 28; 2026: 33.

89 papers.

Sort:

By area:

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  3. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  4. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  5. 2025Kimi K2: Open Agentic Intelligencecited by 523
  6. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 402
  7. 2025Muon is Scalable for LLM Trainingcited by 381
  8. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 283
  9. 2024Tool Learning with Large Language Models: A Surveycited by 221
  10. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  11. 2024Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencentcited by 89
  12. 2024MultiMath: Bridging Visual and Mathematical Reasoning for Large Language Modelscited by 69
38 more
  1. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  2. 2026LLaDA2.1: Speeding Up Text Diffusion via Token Editingcited by 48
  3. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  4. 2025Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Modelscited by 44
  5. 2023LogGPT: Exploring ChatGPT for Log-Based Anomaly Detectioncited by 33
  6. 2023A Fast Optimization View: Reformulating Single Layer Attention in LLM Based on Tensor and SVM Trick, and Solving It in Matrix Multiplication Timecited by 30
  7. 2025Benchmarking Reasoning Robustness in Large Language Modelscited by 30
  8. 2023Who is ChatGPT? Benchmarking LLMs' Psychological Portrayal Using PsychoBenchcited by 28
  9. 2025Hunyuan-TurboS: Advancing Large Language Models through Mamba-Transformer Synergy and Adaptive Chain-of-Thoughtcited by 28
  10. 2024Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detectioncited by 27
  11. 2024Skywork-Math: Data Scaling Laws for Mathematical Reasoning in Large Language Models -- The Story Goes Oncited by 24
  12. 2025DeepTheorem: Advancing LLM Reasoning for Theorem Proving Through Natural Language and Reinforcement Learningcited by 23
  13. 2024Policy Filtration for RLHF to Mitigate Noise in Reward Modelscited by 21
  14. 2025STAR-R1: Spatial TrAnsformation Reasoning by Reinforcing Multimodal LLMscited by 21
  15. 2025WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-trainingcited by 21
  16. 2026Large Language Model Reasoning Failurescited by 20
  17. 2024Causal Evaluation of Language Modelscited by 18
  18. 2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17
  19. 2025Semantic-Space Exploration and Exploitation in RLVR for LLM Reasoningcited by 17
  20. 2025FastMCTS: A Simple Sampling Strategy for Data Synthesiscited by 16
  21. 2026ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMscited by 14
  22. 2024Distilling Reasoning Ability from Large Language Models with Adaptive Thinkingcited by 13
  23. 2025Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend NPUscited by 13
  24. 2025A Survey on Mathematical Reasoning and Optimization with Large Language Modelscited by 12
  25. 2024FineMath: A Fine-Grained Mathematical Evaluation Benchmark for Chinese Large Language Modelscited by 10
  26. 2025Benchmarking Multimodal Mathematical Reasoning with Explicit Visual Dependencycited by 10
  27. 2024OpenEval: Benchmarking Chinese LLMs across Capability, Alignment and Safetycited by 9
  28. 2026Exact Is Easier: Credit Assignment for Cooperative LLM Agentscited by 9
  29. 2026Parallelism and Generation Order in Masked Diffusion Language Models: Limits Today, Potential Tomorrowcited by 8
  30. 2025Orthogonal Finetuning Made Scalablecited by 7
  31. 2026MergeMix: Optimizing Mid-Training Data Mixtures via Learnable Model Mergingcited by 7
  32. 2024Exploring Mathematical Extrapolation of Large Language Models with Synthetic Datacited by 6
  33. 2024Dr.Academy: A Benchmark for Evaluating Questioning Capability in Education for Large Language Modelscited by 6
  34. 2024SIaM: Self-Improving Code-Assisted Mathematical Reasoning of Large Language Modelscited by 6
  35. 2025APTBench: Benchmarking Agentic Potential of Base LLMs During Pre-Trainingcited by 6
  36. 2026FBS: Modeling Native Parallel Reading inside a Transformercited by 6
  37. 2026Ling and Ring 2.6 Technical Report: Efficient and Instant Agentic Intelligence at Trillion-Parameter Scalecited by 6
  38. 2024Leveraging Web-Crawled Data for High-Quality Fine-Tuningcited by 5

50 most cited of 89 All 89 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating Large Language Models on the 2026 Korean CSAT Mathematics Exam: Measuring Mathematical Ability in a Zero-Data-Leakage Setting
  2. 2024CMMaTH: A Chinese Multi-modal Math Skill Evaluation Benchmark for Foundation Models
  3. 2024CMM-Math: A Chinese Multimodal Math Dataset To Evaluate and Enhance the Mathematics Reasoning of Large Multimodal Models
  4. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmetic
  5. 2025Evaluation of LLMs for mathematical problem solving
  6. 2025Large Language Models Don't Make Sense of Word Problems. A Scoping Review from a Mathematics Education Perspective
  7. 2026The CompMath-MCQ Dataset: Are LLMs Ready for Higher-Level Math?
  8. 2024SuperCLUE-Math6: Graded Multi-Step Math Reasoning Benchmark for LLMs in Chinese