mlsanityData to 14 Oct 2026

Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers

arXiv 2510.11370 ↗First posted 13 Oct 2025cs.CL cs.AI cs.LGcited by 72

On the map: Mixture-of-experts → MoE in reinforcement learning and robotics

Abstract (arXiv)

Reinforcement learning (RL) has emerged as a crucial approach for enhancing the capabilities of large language models. However, in Mixture-of-Experts (MoE) models, the routing mechanism often introduces instability, even leading to catastrophic RL training collapse. We analyze the training-inference consistency of MoE models and identify a notable discrepancy in routing behaviors between the two phases. Moreover, even under identical conditions, the routing framework can yield divergent expert selections across repeated forward passes. To address this foundational inconsistency, we propose Rollout Routing Replay (R3), a method that records routing distributions from the inference engine and replays them during training. R3 significantly reduces training-inference policy KL divergence and mitigates extreme discrepancies without compromising training speed. Extensive experiments on various settings confirm that R3 succeeds in stabilizing RL training, preventing collapse and outperforming methods such as GSPO and TIS. We believe this work can offer a new solution for stabilizing RL in MoE models.

Key facts. Cited by 72 papers in the map (data through 14 Oct 2026). Placed in MoE in reinforcement learning and robotics, one of 12 directions in Mixture-of-experts. Builds on 23 of its 37 references that are in the map. First posted on arXiv 13 Oct 2025.

Builds on

23 of 37 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention16 Jun

Other references · 22

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  9. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  11. 2024Mixtral of Expertscited by 1,823
  12. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
10 more
  1. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  2. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 647
  3. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 461
  4. 2025R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agentscited by 158
  5. 2025Geometric-Mean Policy Optimizationcited by 111
  6. 2025Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Modelscited by 109
  7. 2022StableMoE: Stable Routing Strategy for Mixture of Expertscited by 100
  8. 2025On-Policy RL with Optimal Reward Baselinecited by 34
  9. 2025AM-Thinking-v1: Advancing the Frontier of Reasoning at 32B Scalecited by 29
  10. 2025Continual Pre-training of MoEs: How robust is your router?cited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 68.

72 papers.

Sort:

By area:

  1. 2026MiMo-V2-Flash Technical Reportcited by 191
  2. 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practicescited by 41
  3. 2026Rethinking the Trust Region in LLM Reinforcement Learningcited by 37
  4. 2026Intern-S1-Pro: Scientific Multimodal Foundation Model at Trillion Scalecited by 29
  5. 2026Composer 2 Technical Reportcited by 22
  6. 2026Scalable Training of Mixture-of-Experts Models with Megatron Corecited by 19
  7. 2026PyVision-RL: Forging Open Agentic Vision Models via RLcited by 16
  8. 2026FP8-RL: A Practical and Stable Low-Precision Stack for LLM Reinforcement Learningcited by 13
  9. 2026ERNIE 5.0 Technical Reportcited by 11
  10. 2026VESPO: Variational Sequence-Level Soft Policy Optimization for Stable Off-Policy LLM Trainingcited by 11
  11. 2026Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMscited by 11
  12. 2026KDFlow: A User-Friendly and Efficient Knowledge Distillation Framework for Large Language Modelscited by 7
38 more
  1. 2026Relax: An Asynchronous Reinforcement Learning Engine for Omni-Modal Post-Training at Scalecited by 7
  2. 2026Diagnosing Training Inference Mismatch in LLM Reinforcement Learning via a Zero-Mismatch Referencecited by 7
  3. 2026T1: Terminal Agent Reinforcement Learning for Long-Horizon Taskscited by 7
  4. 2026GoLongRL: Capability-Oriented Long Context Reinforcement Learning with Multitask Alignmentcited by 6
  5. 2026PBSD: Privileged Bayesian Self-Distillation for Long-Horizon Credit Assignmentcited by 6
  6. 2026Grouter: Decoupling Routing from Representation for Accelerated MoE Trainingcited by 5
  7. 2026Graph-GRPO: Training Graph Flow Models with Reinforcement Learningcited by 5
  8. 2026Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correctioncited by 5
  9. 2026Stale but Stable: Staleness-Adaptive Trust Regions for Stabilizing Asynchronous Reinforcement Learningcited by 5
  10. 2026Not All LLM Reasoning is Visible in the Chain-of-Thoughtcited by 5
  11. 2026Full-bandwidth transformercited by 5
  12. 2026ZAYA1-8B Technical Reportcited by 4
  13. 2026Intern-S2-Preview: Scientific Agentic Foundation Modelcited by 4
  14. 2026LEGO-RL: Harness-Native Reinforcement Learning for Coding Agentscited by 4
  15. 2025Reinforcement Learning for Chain of Thought Compression with One-Domain-to-All Generalizationcited by 3
  16. 2026Probing RLVR training instability through the lens of objective-level hackingcited by 3
  17. 2026Balanced Aggregation: Understanding and Fixing Aggregation Bias in GRPOcited by 3
  18. 2026ReLibra: Routing-Replay-Guided Load Balancing for MoE Training in Reinforcement Learningcited by 3
  19. 2026ASymPO: Asymmetric-Scale Policy Optimization for Asynchronous LLM Post-Training Without Behavior Informationcited by 3
  20. 2026Score Centering Stabilizes Off-policy Reinforcement Learningcited by 3
  21. 2025RLoop: An Self-Improving Framework for Reinforcement Learning with Iterative Policy Initializationcited by 2
  22. 2025Each Prompt Matters: Scaling Reinforcement Learning Without Wasting Rollouts on Hundred-Billion-Scale MoEcited by 2
  23. 2026Teaching Thinking Models to Reason with Tools: A Full-Pipeline Recipe for Tool-Integrated Reasoningcited by 2
  24. 2026Beyond Trajectory Rewards: Step-level Credit Assignment for Agentic Search via Graph Modelingcited by 2
  25. 2026Harnessing Routing Foresight for Micro-step-level MoE load balancing in RL Post-trainingcited by 2
  26. 2026Turning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignmentcited by 2
  27. 2026MoRSE: Task-Oriented Multi-Agent System with Mixture of Role-Subtask Expertscited by 2
  28. 2026Environment Evolution for Terminal Agentscited by 2
  29. 2026MinT: Managed Infrastructure for Training and Serving Millions of LLMscited by 1
  30. 2026PR2: Predictive Routing Replay for MoE-Based LLM Reinforcement Learningcited by 1
  31. 2026Moebius: Serving Mixture-of-Expert Models with Seamless Runtime Parallelism Switchcited by 1
  32. 2026Macaron-V1: Towards Open Continual Learning with Self-Improvement and Mixture-of-LoRAcited by 1
  33. 2026Miles v0.1: Production-Level Post-Trainingcited by 1
  34. 2026MiMo-V2.6: Scaling Reinforcement Learning Towards Self-Improvementcited by 1
  35. 2026Demystifying Design Choices of Reinforcement Fine-tuning: A Batched Contextual Bandit Learning Perspectivecited by 0
  36. 2026Mixture-of-Experts in Remote Sensing: A Surveycited by 0
  37. 2026Trust the Batch, On- or Off-Policy: Adaptive Policy Optimization for RL Post-Trainingcited by 0
  38. 2026Long Live The Balance: Information Bottleneck Driven Tree-based Policy Optimizationcited by 0

50 most cited of 72 All 72 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Stable and Effective Reinforcement Learning for Mixture-of-Experts
  2. 2026Structuring MoE Expert Selection for Agentic Reinforcement Learning
  3. 2026QUADS: Stabilizing NVFP4 Reinforcement Learning for MoE via QUantization-error Alignment across Dual Sides
  4. 2026ACRL: Adaptive Control of Training-Inference Discrepancy for Stable Reinforcement Learning
  5. 2026A Step Back: Prefix Importance Ratio Stabilizes Policy Optimization
  6. 2026ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples
  7. 2025Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?
  8. 2025Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection