Rethinking Expert Trajectory Utilization in LLM Post-training for Mathematical Reasoning
On the map: LLM reasoning → Reinforcement learning for reasoning
Abstract (arXiv)
Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) dominate the post-training landscape for mathematical reasoning, yet differ fundamentally in their reliance on expert trajectories. To understand the optimal way to harness these trajectories for maximizing performance, we propose the Plasticity-Ceiling Framework. This framework empirically grounds the post-training landscape by decomposing the final performance ceiling into the foundational SFT performance and the subsequent RL plasticity (i.e., the maximum improvement via RL). Through extensive benchmarking, we establish the Sequential SFT-then-RL pipeline as the superior standard, overcoming the stability and premature convergence deficits inherent in synchronized approaches. Furthermore, we derive precise scaling guidelines: (1) Transitioning to RL at the Stable or Mild Overfitting Regime of SFT maximizes the final ceiling by securing a robust SFT foundation with substantial RL plasticity; (2) Refuting the ``Less is More'' hypothesis in SFT-then-RL scaling, we demonstrate that Data Scale determines the primary post-training potential, while Trajectory Difficulty acts as a performance multiplier; and (3) The Minimum Validation Loss of SFT serves as a reliable indicator for selecting the expert trajectories that maximize the ultimate performance ceiling. Our findings provide actionable guidelines for extracting maximum value from expert trajectories.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 46 of its 59 references that are in the map. First posted on arXiv 12 Dec 2025.
Builds on
46 of 59 references.
Closest ancestors · 9
Same direction, newest first
- 2025Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Instead2 Oct
- 2025Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Them13 Jul
- 2025A Practical Two-Stage Recipe for Mathematical LLMs: Maximizing Accuracy with SFT and Efficiency with Reinforcement Learning11 Jul
- 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling2 Jul
- 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning24 Jun
- 2025Learning to Reason under Off-Policy Guidance21 Apr
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
- 2023Let's Verify Step by Step31 May
Other references · 37
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2025Qwen3 Technical Reportcited by 8,820
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2024Qwen2.5 Technical Reportcited by 4,942
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
- 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
- 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
- 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
- 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
- 2025s1: Simple test-time scalingcited by 1,341
- 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
25 more
- 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
- 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
- 2024DeepSeek-V3 Technical Reportcited by 811
- 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
- 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 664
- 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
- 2025LIMO: Less is More for Reasoningcited by 498
- 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 467
- 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 176
- 2025Skywork Open Reasoner 1 Technical Reportcited by 165
- 2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 130
- 2025Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoningcited by 106
- 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 101
- 2024DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solvingcited by 95
- 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 91
- 2023LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Modelscited by 88
- 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 59
- 20251.4 Million Open-Source Distilled Reasoning Dataset to Empower Large Language Model Trainingcited by 49
- 2024Scaling Laws for Predicting Downstream Performance in LLMscited by 48
- 2025Towards a Unified View of Large Language Model Post-Trainingcited by 36
- 2025D3: Diversity, Difficulty, and Dependability-Aware Data Selection for Sample-Efficient LLM Instruction Tuningcited by 21
- 2025Climbing the Ladder of Reasoning: What LLMs Can-and Still Can't-Solve after SFT?cited by 21
- 2025How to Train Your LLM Web Agent: A Statistical Diagnosiscited by 19
- 2025DeepDistill: Enhancing LLM Reasoning Capabilities via Large-Scale Difficulty-Graded Data Trainingcited by 18
- 2025Implicit Reward as the Bridge: A Unified View of SFT and DPO Connectionscited by 17
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- 2025Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning
- 2026Offline Exploration-Aware Fine-Tuning for Long-Chain Mathematical Reasoning
- 2025RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs
- 2026Select and Improve: Understanding the Mechanics of Post-Training for Reasoning
- 2026Understanding the Synergy between SFT, RLVR, and OPD in LLM Post-Training
- 2025Boosting Accuracy and Efficiency of Budget Forcing in LLMs via Reinforcement Learning for Mathematical Reasoning
- 2026From Meta-Thought to Execution: Cognitively Aligned Post-Training for Generalizable and Reliable LLM Reasoning