# What Makes a Good Curriculum? Disentangling the Effects of Data Ordering on LLM Mathematical Reasoning

arXiv: https://arxiv.org/abs/2510.19099 · HTML: https://mlsanity.com/paper/2510.19099/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Mathematical reasoning, one of 17 directions in LLM reasoning.
- Builds on 34 of its 55 references that are in the map.
- First posted on arXiv 21 Oct 2025.

## Placed in

- LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/)

## Builds on (most cited first, 30 of 34)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [STaR: Bootstrapping Reasoning With Reasoning](https://mlsanity.com/paper/2203.14465/) (2022)
- [MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models](https://mlsanity.com/paper/2309.12284/) (2023)
- [Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning](https://mlsanity.com/paper/2506.01939/) (2025)
- [Formal Mathematics Statement Curriculum Learning](https://mlsanity.com/paper/2202.01344/) (2022)
- [Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models](https://mlsanity.com/paper/2307.14430/) (2023)
- [Self-Evolving Curriculum for LLM Reasoning](https://mlsanity.com/paper/2505.14970/) (2025)
- [MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark](https://mlsanity.com/paper/2405.12209/) (2024)
- [DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving](https://mlsanity.com/paper/2407.13690/) (2024)
- [Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning](https://mlsanity.com/paper/2506.06632/) (2025)
- [Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning](https://mlsanity.com/paper/2402.05808/) (2024)
- [Let's Learn Step by Step: Enhancing In-Context Learning Ability with Curriculum Learning](https://mlsanity.com/paper/2402.10738/) (2024)
- [Curriculum-RLAIF: Curriculum Alignment with Reinforcement Learning from AI Feedback](https://mlsanity.com/paper/2505.20075/) (2025)
- [SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learning](https://mlsanity.com/paper/2506.09016/) (2025)
- [Strategic Data Ordering: Enhancing Large Language Model Performance through Curriculum Learning](https://mlsanity.com/paper/2405.07490/) (2024)
- [Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum](https://mlsanity.com/paper/2405.13226/) (2024)
- [Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learning](https://mlsanity.com/paper/2506.11300/) (2025)
- [Easy2Hard-Bench: Standardized Difficulty Labels for Profiling LLM Performance and Generalization](https://mlsanity.com/paper/2409.18433/) (2024)
- [Train Long, Think Short: Curriculum Learning for Efficient Reasoning](https://mlsanity.com/paper/2508.08940/) (2025)
- [On the Noise Robustness of In-Context Learning for Text Generation](https://mlsanity.com/paper/2405.17264/) (2024)
- [CITING: Large Language Models Create Curriculum for Instruction Tuning](https://mlsanity.com/paper/2310.02527/) (2023)
- [Adaptive Ranking-based Sample Selection for Weakly Supervised Class-imbalanced Text Classification](https://mlsanity.com/paper/2210.03092/) (2022)

## Built on by (most cited first, 11 of 11)

- [Behavior Knowledge Merge in Reinforced Agentic Models](https://mlsanity.com/paper/2601.13572/) (2026)
- [Addressing Overthinking in Large Vision-Language Models via Gated Perception-Reasoning Optimization](https://mlsanity.com/paper/2601.04442/) (2026)
- [When Linguistic and Internal Confidence Diverge in Large Language Models](https://mlsanity.com/paper/2608.28382/) (2026)
- [D3: Dynamic Directional Graph-Constrained Data Scheduling for LLM Training](https://mlsanity.com/paper/2605.31164/) (2026)
- [Understanding Curriculum Learning in Large Language Models via Cross-Difficulty Optimization Dynamics](https://mlsanity.com/paper/2608.17268/) (2026)
- [Curriculum Learning as Transport: Understanding Curricula with Wasserstein Geodesics](https://mlsanity.com/paper/2609.09099/) (2026)
- [Which Tokens Should SFT Actually Learn? A Token-Trimming Perspective on Mathematical Reasoning](https://mlsanity.com/paper/2609.09707/) (2026)
- [Rethinking Length-Based Training: Batch Composition and Loss Normalization in Speech Token Language Models](https://mlsanity.com/paper/2609.25890/) (2026)
- [The Free-Recipe Limit: Every Recipe Effect Measures Which Premise of an Idealised Learner Broke](https://mlsanity.com/paper/2609.26160/) (2026)
- [MetaCtrl: Your Large Language Models Can Reason Better and More Concisely with a Metacognitive Controller](https://mlsanity.com/paper/2609.37304/) (2026)
- [Conflicting Supervision Moves Commitment, Not Capability: A 12.29σ arrangement effect that is exactly zero under a convention-agnostic score](https://mlsanity.com/paper/2610.00234/) (2026)

## Abstract (arXiv)

Curriculum learning (CL) - ordering training data from easy to hard - has become a popular strategy for improving reasoning in large language models (LLMs). Yet prior work employs disparate difficulty metrics and training setups, leaving open fundamental questions: When does curriculum help? Which direction - forward or reverse - is better? And does the answer depend on what we measure? We address these questions through a unified offline evaluation framework that decomposes curriculum difficulty into five complementary dimensions: Problem Difficulty, Model Surprisal, Confidence Margin, Predictive Uncertainty, and Decision Variability. Through controlled post-training experiments on mathematical reasoning benchmarks with Llama3.1-8B, Mistral-7B, and Gemma3-4B, we find that (i) no curriculum strategy dominates universally - the relative effectiveness of forward versus reverse CL depends jointly on model capability and task complexity; (ii) even within a single metric, samples at different difficulty levels produce distinct gains depending on task demands; and (iii) task-aligned curricula focus on shaping the model's final representations and generalization, whereas inner-state curricula modulate internal states such as confidence and uncertainty. Our findings challenge the notion of a universal curriculum strategy and offer actionable guidance across model and task regimes, with some metrics indicating that prioritizing decision-uncertain samples can further enhance learning outcomes.