mlsanityData to 14 Oct 2026

Score Centering Stabilizes Off-policy Reinforcement Learning

arXiv 2609.20807 ↗First posted 17 Sep 2026cs.LGcited by 3

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

Reinforcement learning (RL) of large language models is notoriously sensitive to small differences between training and inference engines, often referred to as the training-inference mismatch (TIM). However, completely eliminating TIM is impractical, as it would come at a major cost to rollout efficiency. In this paper, we show that the instability of RL under TIM is primarily caused by drift: a persistent bias between training and inference engines that accumulates with every training step. We derive an additive "score centering" correction term that stabilizes RL under TIM by canceling drift. When training models from 0.6B to 30B parameters, score centering alone matches or outperforms methods based on importance sampling under quantization, with the gap growing as the mismatch becomes more severe. Because the correction is additive, score centering also composes with importance sampling -- their composition outperforms pure importance-sampling baselines in our staleness experiments.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 28 of its 42 references that are in the map. First posted on arXiv 17 Sep 2026.

Builds on

28 of 42 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Jet-RL: Enabling On-Policy FP8 Reinforcement Learning with Unified Training and Rollout Precision Flow20 Jan
  2. 2025Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model21 Oct
  3. 2025PipelineRL: Faster On-policy Reinforcement Learning for Long Sequence Generation23 Sep

Other references · 25

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  7. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  8. 2025Group Sequence Policy Optimizationcited by 835
  9. 2013Black Box Variational Inferencecited by 319
  10. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 250
  11. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 162
  12. 2024Stream of Search (SoS): Learning to Search in Languagecited by 145
13 more
  1. 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 101
  2. 2024Learning Dynamics of LLM Finetuningcited by 95
  3. 2025Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routerscited by 72
  4. 2025Defeating the Training-Inference Mismatch via FP16cited by 66
  5. 2025Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMscited by 61
  6. 2026Rethinking the Trust Region in LLM Reinforcement Learningcited by 37
  7. 2026Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learningcited by 37
  8. 2025On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimizationcited by 31
  9. 2025INTELLECT-2: A Reasoning Model Trained Through Globally Decentralized Reinforcement Learningcited by 25
  10. 2026FP8-RL: A Practical and Stable Low-Precision Stack for LLM Reinforcement Learningcited by 13
  11. 2026Do We Need Adam? Surprisingly Strong and Sparse Reinforcement Learning with SGD in LLMscited by 7
  12. 2026Probing RLVR training instability through the lens of objective-level hackingcited by 3
  13. 2026Adaptive Layerwise Perturbation: Unifying Off-Policy Corrections for LLM RLcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026T5: Twin-Critic Training for Token-Level Thoughts in Reinforcement Mid-Trainingcited by 0
  2. 2026Allspark: Weak to Strong Transfer via Alternating Chain of Thoughtcited by 0
  3. 2026TRACE: Rollout-Guided Quantization-Aware Training for FP4 Reinforcement Learning of MoE Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Rethinking Training-Inference Mismatch in LLM Reinforcement Learning: Where It Arises and How to Correct It
  2. 2026Beyond Precision: Training-Inference Mismatch is an Optimization Problem and Simple LR Scheduling Fixes It
  3. 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
  4. 2026A Step Back: Prefix Importance Ratio Stabilizes Policy Optimization
  5. 2026BRACE: Anchored Bellman-Residual Correction for Stale Critics in Asynchronous RL
  6. 2026Diagnosing Training Inference Mismatch in LLM Reinforcement Learning via a Zero-Mismatch Reference
  7. 2026COPC: Coupled Off-Policy Correction for Asynchronous LLM Reinforcement Learning
  8. 2025Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalization