# Can an Easy-to-Hard Curriculum Make Reasoning Emerge in Small Language Models? Evidence from a Four-Stage Curriculum on GPT-2

arXiv: https://arxiv.org/abs/2505.11643 · HTML: https://mlsanity.com/paper/2505.11643/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 12 of its 14 references that are in the map.
- First posted on arXiv 16 May 2025.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)

## Builds on (most cited first, 12 of 12)

- [Atlas: Few-shot Learning with Retrieval Augmented Language Models](https://mlsanity.com/paper/2208.03299/) (2022)
- [Explainability for Large Language Models: A Survey](https://mlsanity.com/paper/2309.01029/) (2023)
- [CurriculumNet: Weakly Supervised Learning from Large-Scale Web Images](https://mlsanity.com/paper/1808.01097/) (2018)
- [NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questions](https://mlsanity.com/paper/2502.13124/) (2025)
- [Large Language Models and Cognitive Science: A Comprehensive Review of Similarities, Differences, and Challenges](https://mlsanity.com/paper/2409.02387/) (2024)
- [Curriculum learning for language modeling](https://mlsanity.com/paper/2108.02170/) (2021)
- [Strategic Data Ordering: Enhancing Large Language Model Performance through Curriculum Learning](https://mlsanity.com/paper/2405.07490/) (2024)
- [Leveraging the Inductive Bias of Large Language Models for Abstract Textual Reasoning](https://mlsanity.com/paper/2110.02370/) (2021)
- [Curriculum Learning for Vision-and-Language Navigation](https://mlsanity.com/paper/2111.07228/) (2021)
- [Probabilistic Analogical Mapping with Semantic Relation Networks](https://mlsanity.com/paper/2103.16704/) (2021)
- [CogniDual Framework: Self-Training Large Language Models within a Dual-System Theoretical Framework for Improving Cognitive Tasks](https://mlsanity.com/paper/2409.03381/) (2024)
- [Can Separators Improve Chain-of-Thought Prompting?](https://mlsanity.com/paper/2402.10645/) (2024)

## Built on by (most cited first, 1 of 1)

- [Bottom-up Domain-specific Superintelligence: A Reliable Knowledge Graph is What We Need](https://mlsanity.com/paper/2507.13966/) (2025)

## Abstract (arXiv)

We demonstrate that a developmentally ordered curriculum markedly improves reasoning transparency and sample-efficiency in small language models (SLMs). Concretely, we train Cognivolve, a 124 M-parameter GPT-2 model, on a four-stage syllabus that ascends from lexical matching to multi-step symbolic inference and then evaluate it without any task-specific fine-tuning. Cognivolve reaches target accuracy in half the optimization steps of a single-phase baseline, activates an order-of-magnitude more gradient-salient reasoning heads, and shifts those heads toward deeper layers, yielding higher-entropy attention that balances local and long-range context. The same curriculum applied out of order or with optimizer resets fails to reproduce these gains, confirming that progression--not extra compute--drives the effect. We also identify open challenges: final-answer success still lags a conventional run by about 30%, and our saliency probe under-detects verbal-knowledge heads in the hardest stage, suggesting directions for mixed-stage fine-tuning and probe expansion.