Harnessing Routing Foresight for Micro-step-level MoE load balancing in RL Post-training
On the map: Mixture-of-experts → MoE in reinforcement learning and robotics
Abstract (arXiv)
Mixture-of-Experts (MoE) and reinforcement learning (RL) post-training now dominate large language model (LLM) development, yet expert load imbalance remains a critical challenge. Existing load-balancing systems target pre-training by relying on historical step-level statistics. However, these methods fail under the unique workload dynamics of RL post-training: the step-level load is stable, but the tiny batch sizes processed during micro-steps cause severe, high-frequency load fluctuations. We introduce ForeMoE, a micro-step-level load balancing system for MoE RL post-training. Instead of relying on historical statistics, ForeMoE exploits the multi-stage RL pipeline (rollout, recompute, policy update) by using foreseeable routing information from the rollout stage to proactively guide load balancing in the remaining stages. To support frequent per-micro-step reconfiguration, ForeMoE employs a hierarchical planner that decomposes the NP-hard load balancing problem into tractable sub-components, alongside a transfer engine that leverages complementary hardware paths (CPU-assisted and GPU-direct) for overlapped expert transfer. Evaluations on 64 GPUs demonstrate that ForeMoE achieves up to a 1.45× speedup over state-of-the-art RL post-training systems.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in MoE in reinforcement learning and robotics, one of 12 directions in Mixture-of-experts. Builds on 38 of its 62 references that are in the map. First posted on arXiv 10 Jun 2026.
Builds on
38 of 62 references.
Closest ancestors · 2
Same direction, newest first
- 2026ReLibra: Routing-Replay-Guided Load Balancing for MoE Training in Reinforcement Learning9 May
- 2025Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers13 Oct
Other references · 36
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
- 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
- 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
- 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
- 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
- 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
- 2022Mixture-of-Experts with Expert Choice Routingcited by 623
24 more
- 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
- 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
- 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 250
- 2021Hash Layers For Large Sparse Modelscited by 237
- 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
- 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 201
- 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 134
- 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
- 2015Parallelizing the dual revised simplex methodcited by 89
- 2023FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placementcited by 53
- 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
- 2025Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Expertscited by 49
- 2025LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Trainingcited by 42
- 2025History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRLcited by 38
- 2025Laminar: A Scalable Asynchronous RL Post-Training Frameworkcited by 37
- 2025Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learningcited by 25
- 2025CodeRL+: Improving Code Generation via Reinforcement with Execution Semantics Alignmentcited by 24
- 2025Demystifying NCCL: An In-depth Analysis of GPU Communication Protocols and Algorithmscited by 20
- 2025ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systemscited by 18
- 2026The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertisecited by 13
- 2026StaleFlow: Staleness-Aware Data Management for Mitigating Data Skewness in Fully Disaggregated RL Post-Trainingcited by 11
- 2025MATH-Beyond: A Benchmark for RL to Expand Beyond the Base Modelcited by 10
- 2025Beat the long tail: Distribution-Aware Speculative Decoding for RL Trainingcited by 10
- 2025SYMI: Efficient Mixture-of-Experts Training via Model and Optimizer State Decouplingcited by 5
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2025Towards Stable and Effective Reinforcement Learning for Mixture-of-Experts
- 2025RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Training
- 2026A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMs
- 2026PR2: Predictive Routing Replay for MoE-Based LLM Reinforcement Learning
- 2026Scheduling Mixed RL Rollouts Beyond Prefix Locality
- 2026Libra: Efficient Resource Management for Agentic RL Post-Training
- 2025AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training
- 2026Expert-Space Exploration in MoE Reinforcement Learning