mlsanityData to 7 Oct 2026

Diagnosing Training Inference Mismatch in LLM Reinforcement Learning via a Zero-Mismatch Reference

arXiv 2605.14220 ↗First posted 14 May 2026cs.LG cs.AI cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Modern LLM RL systems separate rollout generation from policy optimization. These two stages are expected to produce token probabilities that match exactly. However, implementation differences can make them assign different values to the same sequence under the same model weights, inducing Training-Inference Mismatch (TIM). TIM is difficult to inspect because it is entangled with off-policy drift and common stabilization mechanisms. In this work, we isolate TIM in a zero-mismatch diagnostic setting (VeXact), and show that small token-level numerical disagreements can independently cause training collapse. We further show that TIM changes the effective optimization problem, and identify a set of remedies that could mitigate TIM. Our results suggest that TIM is not benign numerical noise, but a systems-level perturbation that should be treated as a first-order factor in analyzing LLM RL stability.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 30 of its 44 references that are in the map. First posted on arXiv 14 May 2026.

Builds on

30 of 44 references.

References · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  10. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  11. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  12. 2019Root Mean Square Layer Normalizationcited by 1,425
18 more
  1. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  2. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  3. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
  4. 2025Group Sequence Policy Optimizationcited by 788
  5. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  6. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 234
  7. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalizationcited by 233
  8. 2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
  9. 2024Feedback Loops With Language Models Drive In-Context Reward Hackingcited by 70
  10. 2025Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routerscited by 67
  11. 2025Defeating the Training-Inference Mismatch via FP16cited by 62
  12. 2025Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Modelcited by 50
  13. 2025SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agentcited by 44
  14. 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practicescited by 38
  15. 2025Laminar: A Scalable Asynchronous RL Post-Training Frameworkcited by 35
  16. 2025Deterministic Inference across Tensor Parallel Sizes That Eliminates Training-Inference Mismatchcited by 13
  17. 2025Trust Region Masking for Long-Horizon LLM Reinforcement Learningcited by 13
  18. 2026FP8-RL: A Practical and Stable Low-Precision Stack for LLM Reinforcement Learningcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Re-feeding Is Not Replaying: Measuring Replay Noise in Counterfactual Token-Credit Estimationcited by 1
  2. 2026MoE Proxy Models for Low-Cost Failure Reproduction and Diagnosis in LLM RL Post-Trainingcited by 0
  3. 2026Rollout Efficiency in Reinforcement Learning for Reasoning Large Language Models: A Taxonomy and Future Directionscited by 0
  4. 2026Video-HopChain: Multi-Hop Questions and Confidence-Gated Exploration for Video Reasoning Modelscited by 0
  5. 2026FoldAttention: Declared-Reference Softmax for Fast Decode and Deterministic Backwardcited by 0