SFT Overtraining Predicts Rank Inversion via Entropy Collapse Under RLVR
Not on the map. Why?
Abstract (arXiv)
The standard heuristic of selecting the SFT checkpoint with the highest pass@1 for GRPO can fail when SFT compresses the rollout distribution. For binary rewards, the expected within group advantage variance is p(1-p)(g-1)/g; when early GRPO drives p below p^*(g), most groups have identical rewards and provide no group relative signal. We study SFT depth ladders for Qwen2.5-Coder-3B and DeepSeek-Coder-6.7B. We test Qwen2.5-Coder-3B across five depths and three seeds, and DeepSeek-Coder-6.7B across four matched depths and three seeds. On Qwen, pre RL pass@1 rises with SFT depth, but peak GRPO pass@10 falls from 0.806 to 0.481 (3 seed mean, n=20); pre RL entropy is positively associated with the GRPO outcome (ρ=+0.69). On DeepSeek, pass@1 remains far above p^*(8)=0.083, and GRPO outcomes compress rather than invert. A two stage diagnostic, combining pre RL entropy triage with an early GRPO entropy monitor, flags high risk checkpoints and can stop failing runs early. Simple KL to reference regularisation and label smoothing variants do not rescue the collapsed Qwen checkpoint in our setting, suggesting the failure is not a trivial GRPO hyperparameter artefact.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 17 of its 19 references that are in the map. First posted on arXiv 16 Jun 2026.
Builds on
17 of 19 references.
References · 17
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
- 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
- 2024Qwen2.5-Coder Technical Reportcited by 1,151
- 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
- 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
- 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
- 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
- 2023Understanding plasticity in neural networkscited by 170
- 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Modelscited by 134
- 2025KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Codingcited by 115
5 more
- 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 83
- 2023Maintaining Plasticity in Continual Learning via Regenerative Regularizationcited by 63
- 2025Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Insteadcited by 24
- 2026Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learningcited by 10
- 2025Beyond Pass@k: Breadth-Depth Metrics for Reasoning Boundariescited by 8
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation
- 2026Gradient Starvation in Binary-Reward GRPO: Why Group-Mean Centering Fails and Why the Simplest Fix Works
- 2026F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare
- 2026Group Entropy-Controlled Policy Optimization
- 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic
- 2026When Gradient Importance Lies: Adaptive LoRA Rank Allocation Fails Under GRPO
- 2026HiLoRe: What to Store, Compress, or Recompute for Efficient GRPO Training
- 2025EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for Advantage Diversity