# From Next-Token to Mathematics: The Learning Dynamics of Mathematical Reasoning in Language Models

arXiv: https://arxiv.org/abs/2407.00900 · HTML: https://mlsanity.com/paper/2407.00900/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Mathematical reasoning, one of 17 directions in LLM reasoning.
- Builds on 4 of its 24 references that are in the map.
- First posted on arXiv 1 Jul 2024.

## Placed in

- LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/)

## Builds on (most cited first, 4 of 4)

- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [Large Language Models are Inconsistent and Biased Evaluators](https://mlsanity.com/paper/2405.01724/) (2024)
- [LLM360: Towards Fully Transparent Open-Source LLMs](https://mlsanity.com/paper/2312.06550/) (2023)

## Built on by (most cited first, 6 of 6)

- [MathGAP: Out-of-Distribution Evaluation on Problems with Arbitrarily Complex Proofs](https://mlsanity.com/paper/2410.13502/) (2024)
- [What do Language Models Learn and When? The Implicit Curriculum Hypothesis](https://mlsanity.com/paper/2604.08510/) (2026)
- [Mathfish: Evaluating Language Model Math Reasoning via Grounding in Educational Curricula](https://mlsanity.com/paper/2408.04226/) (2024)
- [A Matter of Interest: Understanding Interestingness of Math Problems in Humans and Language Models](https://mlsanity.com/paper/2511.08548/) (2025)
- [Adaptive Problem Generation via Symbolic Representations](https://mlsanity.com/paper/2602.19187/) (2026)
- [LittleLearner: Language Models Under Pedagogically Controlled Knowledge Exposure](https://mlsanity.com/paper/2608.13545/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) solely trained on next-token prediction learn to solve a wide range of problems involving mathematical reasoning. But how does this ability evolve during training? We show the first analysis of how mathematical reasoning abilities of several open-weight LLMs develop during pre-training and post-training. To this end, we construct MathCAMPS, a synthetic dataset of novel mathematical reasoning problems grounded in 44 fine-grained skills taken from the Common Core curriculum from K to 8th grades. In one experiment, we show that mathematical skills are learned during pre-training in an order that measurably correlates with the human-designed curriculum, even though training data are randomly ordered. We also show a detailed analysis of which mathematical abilities benefit from instruction tuning, a widely used post-training method and, in contrast, which skills suffer. Our work paves the way for an empirical understanding of LLM training dynamics in relation to reasoning.