ACRL: Adaptive Control of Training-Inference Discrepancy for Stable Reinforcement Learning
On the map: Deep reinforcement learning methods → RL training of language and agent models
Abstract (arXiv)
Reinforcement Learning (RL) training for Large Language Models (LLMs) often suffers from instability due to the discrepancy between training and inference. This training-inference discrepancy stems from two primary factors: an architectural separation between training and inference engines, and the use of low-precision quantization in inference versus higher-precision computation in training. To address training instability issues caused by high training-inference discrepancy, we present the principles and methods for its adaptive control. We propose Adaptive Control Reinforcement Learning (ACRL), which adaptively maintains the training-inference discrepancy within a reasonable range to ensure stable RL training. Beyond stabilization, ACRL inherently increases policy entropy, thereby enhancing exploration and improving accuracy. The experimental results show that when the inference engine utilizes FP8 quantization, ACRL consistently maintains the training-inference discrepancy within a reasonable range and stabilizes RL training. Furthermore, ACRL not only matches the accuracy of the BF16 baseline but also outperforms importance sampling (IS) fixes.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 16 of its 25 references that are in the map. First posted on arXiv 27 Jul 2026.
Builds on
16 of 25 references.
References · 16
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
- 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
- 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
- 2025Group Sequence Policy Optimizationcited by 835
- 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 263
- 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 187
- 2018Understanding the impact of entropy on policy optimizationcited by 176
- 2022FP8 Quantization: The Power of the Exponentcited by 88
4 more
- 2025Defeating the Training-Inference Mismatch via FP16cited by 66
- 2025Klear-Reasoner: Advancing Reasoning Capability via Gradient-Preserving Clipping Policy Optimizationcited by 39
- 2025CE-GPPO: Coordinating Entropy via Gradient-Preserving Clipping Policy Optimization in Reinforcement Learningcited by 23
- 2025Recipes for Pre-training LLMs with MXFP8cited by 22
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
- 2026Reformulate LLM Reinforcement Learning for Efficient Training under Black-box Discrepancy
- 2026AIS: Adaptive Importance Sampling for Quantized RL
- 2025On Predictability of Reinforcement Learning Dynamics for Large Language Models
- 2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
- 2025SuperRL: Reinforcement Learning with Supervision to Boost Language Model Reasoning
- 2026Score Centering Stabilizes Off-policy Reinforcement Learning
- 2025RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments