mlsanityData to 14 Oct 2026

Complementing reinforcement learning with SFT through logit averaging in the post training of LLMs

arXiv 2605.20555 ↗First posted 19 May 2026cs.LG cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

We introduce a novel method that averages the logits of a frozen reference policy (e.g., SFT) and a trainable policy, and incorporate the method into Group Relative Policy Optimization (GRPO). In contrast to Reinforcement Learning with Verifiable Rewards (RLVR) methods, our proposal does not involve a Kullback Leibler (KL) regularization or critic; the trainable policy and the reference anchor are coupled through the logit averaging structure to leverage the reasoning expertise of the trainable policy while maintaining the formatting advantage of SFT. Our method is evaluated on MATH, cn-k12, and MMLU, and the results show a higher accuracy or at least comparable accuracy relative to the canonical KL-regularized GRPO.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 16 of its 19 references that are in the map. First posted on arXiv 19 May 2026.

Builds on

16 of 19 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  2. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 14

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2015Trust Region Policy Optimizationcited by 2,580
  8. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  9. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  10. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 441
  11. 2026Reinforcement Learning via Self-Distillationcited by 332
  12. 2026Self-Distillation Enables Continual Learningcited by 247
2 more
  1. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 135
  2. 2024Tuning Language Models by Proxycited by 89

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification
  2. 2025Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training
  3. 2026Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective
  4. 2026Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning
  5. 2026RRPO: Reference-Relative Policy Optimization with Stratified Conditional Rollouts
  6. 2025Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning
  7. 2025Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning
  8. 2026Breaking textitWinner-Takes-All: Cooperative Policy Optimization Improves Diverse LLM Reasoning