mlsanityData to 14 Oct 2026

Rethinking Expert Trajectory Utilization in LLM Post-training for Mathematical Reasoning

arXiv 2512.11470 ↗First posted 12 Dec 2025cs.LG cs.CLcited by 2

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) dominate the post-training landscape for mathematical reasoning, yet differ fundamentally in their reliance on expert trajectories. To understand the optimal way to harness these trajectories for maximizing performance, we propose the Plasticity-Ceiling Framework. This framework empirically grounds the post-training landscape by decomposing the final performance ceiling into the foundational SFT performance and the subsequent RL plasticity (i.e., the maximum improvement via RL). Through extensive benchmarking, we establish the Sequential SFT-then-RL pipeline as the superior standard, overcoming the stability and premature convergence deficits inherent in synchronized approaches. Furthermore, we derive precise scaling guidelines: (1) Transitioning to RL at the Stable or Mild Overfitting Regime of SFT maximizes the final ceiling by securing a robust SFT foundation with substantial RL plasticity; (2) Refuting the ``Less is More'' hypothesis in SFT-then-RL scaling, we demonstrate that Data Scale determines the primary post-training potential, while Trajectory Difficulty acts as a performance multiplier; and (3) The Minimum Validation Loss of SFT serves as a reliable indicator for selecting the expert trajectories that maximize the ultimate performance ceiling. Our findings provide actionable guidelines for extracting maximum value from expert trajectories.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 46 of its 59 references that are in the map. First posted on arXiv 12 Dec 2025.

Builds on

46 of 59 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Instead2 Oct
  2. 2025Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Them13 Jul
  3. 2025A Practical Two-Stage Recipe for Mathematical LLMs: Maximizing Accuracy with SFT and Efficiency with Reinforcement Learning11 Jul
  4. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling2 Jul
  5. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning24 Jun
  6. 2025Learning to Reason under Off-Policy Guidance21 Apr
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  8. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  9. 2023Let's Verify Step by Step31 May

Other references · 37

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  8. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  9. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  10. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  11. 2025s1: Simple test-time scalingcited by 1,341
  12. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
25 more
  1. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  2. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
  3. 2024DeepSeek-V3 Technical Reportcited by 811
  4. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  5. 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 664
  6. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  7. 2025LIMO: Less is More for Reasoningcited by 498
  8. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 467
  9. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 176
  10. 2025Skywork Open Reasoner 1 Technical Reportcited by 165
  11. 2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 130
  12. 2025Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoningcited by 106
  13. 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 101
  14. 2024DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solvingcited by 95
  15. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 91
  16. 2023LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Modelscited by 88
  17. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 59
  18. 20251.4 Million Open-Source Distilled Reasoning Dataset to Empower Large Language Model Trainingcited by 49
  19. 2024Scaling Laws for Predicting Downstream Performance in LLMscited by 48
  20. 2025Towards a Unified View of Large Language Model Post-Trainingcited by 36
  21. 2025D3: Diversity, Difficulty, and Dependability-Aware Data Selection for Sample-Efficient LLM Instruction Tuningcited by 21
  22. 2025Climbing the Ladder of Reasoning: What LLMs Can-and Still Can't-Solve after SFT?cited by 21
  23. 2025How to Train Your LLM Web Agent: A Statistical Diagnosiscited by 19
  24. 2025DeepDistill: Enhancing LLM Reasoning Capabilities via Large-Scale Difficulty-Graded Data Trainingcited by 18
  25. 2025Implicit Reward as the Bridge: A Unified View of SFT and DPO Connectionscited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
  2. 2025Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning
  3. 2026Offline Exploration-Aware Fine-Tuning for Long-Chain Mathematical Reasoning
  4. 2025RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs
  5. 2026Select and Improve: Understanding the Mechanics of Post-Training for Reasoning
  6. 2026Understanding the Synergy between SFT, RLVR, and OPD in LLM Post-Training
  7. 2025Boosting Accuracy and Efficiency of Budget Forcing in LLMs via Reinforcement Learning for Mathematical Reasoning
  8. 2026From Meta-Thought to Execution: Cognitively Aligned Post-Training for Generalizable and Reliable LLM Reasoning