mlsanityData to 14 Oct 2026

Climbing the Ladder of Reasoning: What LLMs Can-and Still Can't-Solve after SFT?

arXiv 2504.11741 ↗First posted 16 Apr 2025cs.AI cs.CL cs.LGcited by 21

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Recent supervised fine-tuning (SFT) approaches have significantly improved language models' performance on mathematical reasoning tasks, even when models are trained at a small scale. However, the specific capabilities enhanced through such fine-tuning remain poorly understood. In this paper, we conduct a detailed analysis of model performance on the AIME24 dataset to understand how reasoning capabilities evolve. We discover a ladder-like structure in problem difficulty, categorize questions into four tiers (Easy, Medium, Hard, and Extremely Hard (Exh)), and identify the specific requirements for advancing between tiers. We find that progression from Easy to Medium tier requires adopting an R1 reasoning style with minimal SFT (500-1K instances), while Hard-level questions suffer from frequent model's errors at each step of the reasoning chain, with accuracy plateauing at around 65% despite logarithmic scaling. Exh-level questions present a fundamentally different challenge; they require unconventional problem-solving skills that current models uniformly struggle with. Additional findings reveal that carefully curated small-scale datasets offer limited advantage-scaling dataset size proves far more effective. Our analysis provides a clearer roadmap for advancing language model capabilities in mathematical reasoning.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 10 of its 19 references that are in the map. First posted on arXiv 16 Apr 2025.

Builds on

10 of 19 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025LIMO: Less is More for Reasoning5 Feb
  2. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers29 Feb

Other references · 8

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2025s1: Simple test-time scalingcited by 1,341
  5. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 377
  6. 2025LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!cited by 132
  7. 2024OpenR: An Open Source Framework for Advanced Reasoning with Large Language Modelscited by 94
  8. 2025An Empirical Study on Eliciting and Improving R1-like Reasoning Modelscited by 78

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 17; 2026: 4.

21 papers.

Sort:

By area:

  1. 2025WebSailor: Navigating Super-human Reasoning for Web Agentcited by 229
  2. 2025OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalizationcited by 50
  3. 2025NaturalThoughts: Selecting and Distilling Reasoning Traces for General Reasoning Taskscited by 32
  4. 2025Base Models Know How to Reason, Thinking Models Learn Whencited by 27
  5. 2025Synthetic Data RL: Task Definition Is All You Needcited by 15
  6. 2025The Synergy Dilemma of Long-CoT SFT and RL: Investigating Post-Training Techniques for Reasoning VLMscited by 12
  7. 2025Why Less is More (Sometimes): A Theory of Data Curationcited by 8
  8. 2025How and Why LLMs Generalize: A Fine-Grained Analysis of LLM Reasoning from Cognitive Behaviors to Low-Level Patternscited by 8
  9. 2025PATS: Process-Level Adaptive Thinking Mode Switchingcited by 6
  10. 2026Locally Confident, Globally Stuck: The Quality-Exploration Dilemma in Diffusion Language Modelscited by 5
  11. 2025A Survey of Deep Learning for Geometry Problem Solvingcited by 4
  12. 2025Critique-Guided Distillation for Robust Reasoning via Refinementcited by 2
9 more
  1. 2025Rethinking the Unsolvable: When In-Context Search Meets Test-Time Scalingcited by 2
  2. 2025MENTOR: Reinforcement Learning via Flexible Teacher-Optimized Rewards for Tool-Use Distillationcited by 2
  3. 2025Boosting Accuracy and Efficiency of Budget Forcing in LLMs via Reinforcement Learning for Mathematical Reasoningcited by 2
  4. 2025Rethinking Expert Trajectory Utilization in LLM Post-training for Mathematical Reasoningcited by 2
  5. 2025TimeHC-RL: Temporal-aware Hierarchical Cognitive Reinforcement Learning for Enhancing LLMs' Social Intelligencecited by 0
  6. 2025Input-Time Scaling: Adding Noise and Irrelevance into Less-Is-More Drastically Improves Reasoning Performance and Efficiencycited by 0
  7. 2026Metacognitive Behavioral Tuning of Large Language Models for Multi-Hop Question Answeringcited by 0
  8. 2026Transformer See, Transformer Do: Copying as an Intermediate Step in Learning Analogical Reasoningcited by 0
  9. 2026Characterizing Model-Native Skillscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models
  2. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Models
  3. 2025Advancing Mathematical Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages
  4. 2024An Empirical Study of Data Ability Boundary in LLMs' Math Reasoning
  5. 2025DeepDistill: Enhancing LLM Reasoning Capabilities via Large-Scale Difficulty-Graded Data Training
  6. 2024Unleashing LLM Reasoning Capability via Scalable Question Synthesis from Scratch
  7. 2026Reasoning Quality Emerges Early: Data Curation for Reasoning Models
  8. 2025Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning