mlsanityData to 13 Oct 2026

Can an Easy-to-Hard Curriculum Make Reasoning Emerge in Small Language Models? Evidence from a Four-Stage Curriculum on GPT-2

arXiv 2505.11643 ↗First posted 16 May 2025cs.CLcited by 1

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

We demonstrate that a developmentally ordered curriculum markedly improves reasoning transparency and sample-efficiency in small language models (SLMs). Concretely, we train Cognivolve, a 124 M-parameter GPT-2 model, on a four-stage syllabus that ascends from lexical matching to multi-step symbolic inference and then evaluate it without any task-specific fine-tuning. Cognivolve reaches target accuracy in half the optimization steps of a single-phase baseline, activates an order-of-magnitude more gradient-salient reasoning heads, and shifts those heads toward deeper layers, yielding higher-entropy attention that balances local and long-range context. The same curriculum applied out of order or with optimizer resets fails to reproduce these gains, confirming that progression--not extra compute--drives the effect. We also identify open challenges: final-answer success still lags a conventional run by about 30%, and our saliency probe under-detects verbal-knowledge heads in the hardest stage, suggesting directions for mixed-stage fine-tuning and probe expansion.

Key facts. Cited by 1 paper in the map (data through 13 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 12 of its 14 references that are in the map. First posted on arXiv 16 May 2025.

Builds on

12 of 14 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Can Separators Improve Chain-of-Thought Prompting?16 Feb

Other references · 11

  1. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 947
  2. 2023Explainability for Large Language Models: A Surveycited by 333
  3. 2018CurriculumNet: Weakly Supervised Learning from Large-Scale Web Imagescited by 126
  4. 2025NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questionscited by 81
  5. 2024Large Language Models and Cognitive Science: A Comprehensive Review of Similarities, Differences, and Challengescited by 49
  6. 2021Curriculum learning for language modelingcited by 32
  7. 2024Strategic Data Ordering: Enhancing Large Language Model Performance through Curriculum Learningcited by 28
  8. 2021Leveraging the Inductive Bias of Large Language Models for Abstract Textual Reasoningcited by 18
  9. 2021Curriculum Learning for Vision-and-Language Navigationcited by 18
  10. 2021Probabilistic Analogical Mapping with Semantic Relation Networkscited by 14
  11. 2024CogniDual Framework: Self-Training Large Language Models within a Dual-System Theoretical Framework for Improving Cognitive Taskscited by 5

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Bottom-up Domain-specific Superintelligence: A Reliable Knowledge Graph is What We Needcited by 13

Similar, not cited

Close in wording, no citation either way

  1. 2026Rethinking Easy-to-Hard: Limits of Curriculum Learning in Post-Training for Deductive Reasoning
  2. 2026A Systematic Study of Small Language Models on Abstract Reasoning Tasks
  3. 2026Knowing When Thinking Is Not Enough: Teaching Small Reasoning Models to Reason Beyond Their Parametric Knowledge
  4. 2025Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning
  5. 2026Learning What to Learn: Stage-Specific Data Sets for SFT-then-RL in Small Language Model Reasoning
  6. 2026Batched Contextual Reinforcement: A Task-Scaling Law for Efficient Reasoning
  7. 2025SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought
  8. 2026From Atoms to Chains: Divergence-Guided Reasoning Curriculum for Unlabeled LLM Domain Adaptation