mlsanityData to 14 Oct 2026

Defeating the Training-Inference Mismatch via FP16

arXiv 2510.26788 ↗First posted 30 Oct 2025cs.LG cs.AI cs.CLcited by 66

Not on the map. Why?

Cited by 23 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Reinforcement learning (RL) fine-tuning of large language models (LLMs) often suffers from instability due to the numerical mismatch between the training and inference policies. While prior work has attempted to mitigate this issue through algorithmic corrections or engineering alignments, we show that its root cause lies in the floating point precision itself. The widely adopted BF16, despite its large dynamic range, introduces large rounding errors that breaks the consistency between training and inference. In this work, we demonstrate that simply reverting to FP16 effectively eliminates this mismatch. The change is simple, fully supported by modern frameworks with only a few lines of code change, and requires no modification to the model architecture or learning algorithm. Our results suggest that using FP16 uniformly yields more stable optimization, faster convergence, and stronger performance across diverse tasks, algorithms and frameworks. We hope these findings motivate a broader reconsideration of precision trade-offs in RL fine-tuning.

Key facts. Cited by 66 papers in the map (data through 14 Oct 2026). Builds on 25 of its 42 references that are in the map. First posted on arXiv 30 Oct 2025.

Builds on

25 of 42 references.

References · 25

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  9. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  11. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  12. 2017Mixed Precision Trainingcited by 1,156
13 more
  1. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  2. 2025Group Sequence Policy Optimizationcited by 835
  3. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  4. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 511
  5. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 485
  6. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 263
  7. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 232
  8. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scalingcited by 113
  9. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspectivecited by 76
  10. 2025Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Modelcited by 55
  11. 2023Zero Bubble Pipeline Parallelismcited by 40
  12. 2025Optimizing Anytime Reasoning via Budget Relative Policy Optimizationcited by 27
  13. 2025Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoningcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 57.

66 papers.

Sort:

By area:

  1. 2026Vision-DeepResearch: Incentivizing DeepResearch Capability in Multimodal Large Language Modelscited by 37
  2. 2026Rethinking the Trust Region in LLM Reinforcement Learningcited by 37
  3. 2025SofT-GRPO: Surpassing Discrete-Token LLM Reinforcement Learning via Gumbel-Reparameterized Soft-Thinking Policy Optimizationcited by 21
  4. 2026RAGEN-2: Reasoning Collapse in Agentic RLcited by 20
  5. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  6. 2025Deterministic Inference across Tensor Parallel Sizes That Eliminates Training-Inference Mismatchcited by 14
  7. 2026Future-KL Regularized GRPO: Process-Level Credit Assignment from f-Divergence Regularizationcited by 12
  8. 2026Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMscited by 11
  9. 2026Entropy-Preserving Reinforcement Learningcited by 11
  10. 2026Vero: An Open RL Recipe for General Visual Reasoningcited by 11
  11. 2025Youtu-LLM: Unlocking the Native Agentic Potential for Lightweight Large Language Modelscited by 10
  12. 2026LLMs Can Learn to Reason Via Off-Policy RLcited by 10
38 more
  1. 2026Beyond Precision: Training-Inference Mismatch is an Optimization Problem and Simple LR Scheduling Fixes Itcited by 9
  2. 2026KDFlow: A User-Friendly and Efficient Knowledge Distillation Framework for Large Language Modelscited by 7
  3. 2026QaRL: Rollout-Aligned Quantization-Aware RL for Fast and Stable Training under Training--Inference Mismatchcited by 7
  4. 2026Diagnosing Training Inference Mismatch in LLM Reinforcement Learning via a Zero-Mismatch Referencecited by 7
  5. 2026How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learningcited by 7
  6. 2026Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Modelscited by 6
  7. 2026ARFBench: Benchmarking Time Series Question Answering Ability for Software Incident Responsecited by 6
  8. 2026GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Modelscited by 6
  9. 2026Hidden Reliability Risks in Large Language Models: Systematic Identification of Precision-Induced Output Disagreementscited by 5
  10. 2026Sparrow: Sparse Rollout for Stable and Efficient Long-context RL of Large Language Modelscited by 5
  11. 2026Rethinking the Divergence Regularization in LLM RLcited by 5
  12. 2025INTELLECT-3: Technical Reportcited by 4
  13. 2026CLEANER: Self-Purified Trajectories Boost Agentic Reinforcement Learningcited by 4
  14. 2026On the Optimal Reasoning Length for RL-Trained Language Modelscited by 4
  15. 2026ZAYA1-8B Technical Reportcited by 4
  16. 2026vla.cpp: A Unified Inference Runtime for Vision-Language-Action Modelscited by 4
  17. 2026Pythagoras-Prover: Advancing Efficient Formal Proving via Augmented Lean Formalisationcited by 4
  18. 2026Best Practice Critic Optimizationcited by 4
  19. 2025Whatever Remains Must Be True: Filtering Drives Reasoning in LLMs, Shaping Diversitycited by 3
  20. 2026Adaptive Layerwise Perturbation: Unifying Off-Policy Corrections for LLM RLcited by 3
  21. 2026QUADS: Stabilizing NVFP4 Reinforcement Learning for MoE via QUantization-error Alignment across Dual Sidescited by 3
  22. 2026Score Centering Stabilizes Off-policy Reinforcement Learningcited by 3
  23. 2025PLUTO-4: Frontier Pathology Foundation Modelscited by 2
  24. 2025Each Prompt Matters: Scaling Reinforcement Learning Without Wasting Rollouts on Hundred-Billion-Scale MoEcited by 2
  25. 2026BubbleSpec: Turning Long-Tail Bubbles into Speculative Rollout Drafts for Synchronous Reinforcement Learningcited by 2
  26. 2026Demystifying Numerical Instability in LLM Inference: Achieving Reproducible Inference for Mission-Critical Tasks with HEALcited by 2
  27. 2025RLAX: Large-Scale, Distributed Reinforcement Learning for Large Language Models on TPUscited by 1
  28. 2026PCL-Reasoner-V1.5: Advancing Math Reasoning with Offline Reinforcement Learningcited by 1
  29. 2026PrAg-PO: Prompt Augmented Policy Optimization for Robust and Diverse Mathematical Reasoningcited by 1
  30. 2026Online Causal Kalman Filtering for Stable and Effective Policy Optimizationcited by 1
  31. 2026Cost-Aware Learningcited by 1
  32. 2026Multi-Step Likelihood-Ratio Correction for Reinforcement Learning with Verifiable Rewardscited by 1
  33. 2026Re-feeding Is Not Replaying: Measuring Replay Noise in Counterfactual Token-Credit Estimationcited by 1
  34. 2026The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learningcited by 1
  35. 2026ACRL: Adaptive Control of Training-Inference Discrepancy for Stable Reinforcement Learningcited by 1
  36. 2026Reading the Gate, Not the Interference: Output-Side Interference Measurement Does Not Track Merge Collapsecited by 1
  37. 2026DenseMLLM: Standard Multimodal LLMs for Dense Predictioncited by 0
  38. 2026Learning to Detect Language Model Training Data via Active Reconstructioncited by 0

50 most cited of 66 All 66 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Them
  2. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models
  3. 2025RL Is Neither a Panacea Nor a Mirage: Understanding Supervised vs. Reinforcement Learning Fine-Tuning for LLMs
  4. 2026The Optimal Token Baseline: Variance Reduction for Long-Horizon LLM-RL
  5. 2025Reinforcement Learning Fine-Tunes a Sparse Subnetwork in Large Language Models
  6. 2026Jet-RL: Enabling On-Policy FP8 Reinforcement Learning with Unified Training and Rollout Precision Flow
  7. 2026FP8-RL: A Practical and Stable Low-Precision Stack for LLM Reinforcement Learning
  8. 2026A Step Back: Prefix Importance Ratio Stabilizes Policy Optimization