# Beyond Precision: Training-Inference Mismatch is an Optimization Problem and Simple LR Scheduling Fixes It

arXiv: https://arxiv.org/abs/2602.01826 · HTML: https://mlsanity.com/paper/2602.01826/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 31 references that are in the map.
- First posted on arXiv 2 Feb 2026.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?](https://mlsanity.com/paper/2504.13837/) (2025)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Group Sequence Policy Optimization](https://mlsanity.com/paper/2507.18071/) (2025)
- [PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel](https://mlsanity.com/paper/2304.11277/) (2023)
- [The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models](https://mlsanity.com/paper/2505.22617/) (2025)
- [A Long Way to Go: Investigating Length Correlations in RLHF](https://mlsanity.com/paper/2310.03716/) (2023)
- [MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention](https://mlsanity.com/paper/2506.13585/) (2025)
- [Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning](https://mlsanity.com/paper/2502.14768/) (2025)
- [Defeating the Training-Inference Mismatch via FP16](https://mlsanity.com/paper/2510.26788/) (2025)
- [Stable Reinforcement Learning for Efficient Reasoning](https://mlsanity.com/paper/2505.18086/) (2025)
- [Shorten After You're Right: Lazy Length Penalties for Reasoning RL](https://mlsanity.com/paper/2505.12284/) (2025)
- [MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions](https://mlsanity.com/paper/2410.02743/) (2024)
- [Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoning](https://mlsanity.com/paper/2510.19338/) (2025)

## Built on by (most cited first, 9 of 9)

- [RAGEN-2: Reasoning Collapse in Agentic RL](https://mlsanity.com/paper/2604.06268/) (2026)
- [The Optimal Token Baseline: Variance Reduction for Long-Horizon LLM-RL](https://mlsanity.com/paper/2602.07078/) (2026)
- [Joint Training of Multi-Token Prediction in Reinforcement Learning via Optimal Coefficient Calibration](https://mlsanity.com/paper/2605.28184/) (2026)
- [The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning](https://mlsanity.com/paper/2606.29526/) (2026)
- [Off-Policy Value-Based Reinforcement Learning for Large Language Models](https://mlsanity.com/paper/2603.23355/) (2026)
- [Reformulate LLM Reinforcement Learning for Efficient Training under Black-box Discrepancy](https://mlsanity.com/paper/2606.08779/) (2026)
- [MemoryWalker: Stop Training Agents on Contexts They Never Saw](https://mlsanity.com/paper/2609.00865/) (2026)
- [Rethinking Training-Inference Mismatch in LLM Reinforcement Learning: Where It Arises and How to Correct It](https://mlsanity.com/paper/2609.32444/) (2026)
- [Measuring and Reducing Cross-Vendor Mismatch in Language Models](https://mlsanity.com/paper/2610.05458/) (2026)

## Abstract (arXiv)

Reinforcement Learning (RL) for training Large Language Models is notoriously unstable. While recent studies attribute this to "training inference mismatch stemming" from inconsistent hybrid engines, standard remedies, such as Importance Sampling, might fail during extended training runs. In this work, we analyze this instability through the lens of optimization, demonstrating that gradient noise and training-inference mismatch escalate in tandem as training progresses. Meanwhile, we find that the mismatch can be effectively suppressed by shrinking the update size. Taken together, we deduce that the mismatch is not merely a static numerical discrepancy, but a dynamic failure coupled with the model's optimization. Based on this insight, we propose a simple yet effective solution: a specialized Learning Rate (LR) scheduler. Instead of pre-defined decay schedule in traditional LR scheduler, our method dynamically triggers LR decay based on response length, which we identify as a reliable early-warning signal for impending instability. Empirical evidence suggests that by reducing the learning rate as gradient noise rises, we can consistently stabilize RL training and keep the training-inference mismatch at a safe level.