# Harnessing Routing Foresight for Micro-step-level MoE load balancing in RL Post-training

arXiv: https://arxiv.org/abs/2606.11867 · HTML: https://mlsanity.com/paper/2606.11867/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in MoE in reinforcement learning and robotics, one of 12 directions in Mixture-of-experts.
- Builds on 38 of its 62 references that are in the map.
- First posted on arXiv 10 Jun 2026.

## Placed in

- Mixture-of-experts / [MoE in reinforcement learning and robotics](https://mlsanity.com/moe/moe-in-reinforcement-learning-and-robotics/)

## Builds on (most cited first, 30 of 38)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://mlsanity.com/paper/2112.06905/) (2021)
- [PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel](https://mlsanity.com/paper/2304.11277/) (2023)
- [Mixture-of-Experts with Expert Choice Routing](https://mlsanity.com/paper/2202.09368/) (2022)
- [Ray: A Distributed Framework for Emerging AI Applications](https://mlsanity.com/paper/1712.05889/) (2017)
- [BASE Layers: Simplifying Training of Large, Sparse Models](https://mlsanity.com/paper/2103.16716/) (2021)
- [AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning](https://mlsanity.com/paper/2505.24298/) (2025)
- [Hash Layers For Large Sparse Models](https://mlsanity.com/paper/2106.04426/) (2021)
- [A Survey on Mixture of Experts in Large Language Models](https://mlsanity.com/paper/2407.06204/) (2024)
- [Tutel: Adaptive Mixture-of-Experts at Scale](https://mlsanity.com/paper/2206.03382/) (2022)
- [Memory-Efficient Pipeline-Parallel DNN Training](https://mlsanity.com/paper/2006.09503/) (2020)
- [A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applications](https://mlsanity.com/paper/2503.07137/) (2025)
- [Parallelizing the dual revised simplex method](https://mlsanity.com/paper/1503.01889/) (2015)
- [Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers](https://mlsanity.com/paper/2510.11370/) (2025)
- [FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement](https://mlsanity.com/paper/2304.03946/) (2023)
- [A Survey on Large Language Models for Mathematical Reasoning](https://mlsanity.com/paper/2506.08446/) (2025)
- [Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Experts](https://mlsanity.com/paper/2502.19811/) (2025)
- [LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Training](https://mlsanity.com/paper/2505.24034/) (2025)
- [History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRL](https://mlsanity.com/paper/2508.18588/) (2025)
- [Laminar: A Scalable Asynchronous RL Post-Training Framework](https://mlsanity.com/paper/2510.12633/) (2025)
- [Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learning](https://mlsanity.com/paper/2511.14617/) (2025)
- [CodeRL+: Improving Code Generation via Reinforcement with Execution Semantics Alignment](https://mlsanity.com/paper/2510.18471/) (2025)

## Built on by (most cited first, 2 of 2)

- [ScienceIDE: Turning World's Scientific Codebase into Agent Learnable Environments](https://mlsanity.com/paper/2609.19134/) (2026)
- [RoutePack: Expert Placement and Attention-Aware Data Packing for MoE Reinforcement Learning](https://mlsanity.com/paper/2608.12146/) (2026)

## Abstract (arXiv)

Mixture-of-Experts (MoE) and reinforcement learning (RL) post-training now dominate large language model (LLM) development, yet expert load imbalance remains a critical challenge. Existing load-balancing systems target pre-training by relying on historical step-level statistics. However, these methods fail under the unique workload dynamics of RL post-training: the step-level load is stable, but the tiny batch sizes processed during micro-steps cause severe, high-frequency load fluctuations. We introduce ForeMoE, a micro-step-level load balancing system for MoE RL post-training. Instead of relying on historical statistics, ForeMoE exploits the multi-stage RL pipeline (rollout, recompute, policy update) by using foreseeable routing information from the rollout stage to proactively guide load balancing in the remaining stages. To support frequent per-micro-step reconfiguration, ForeMoE employs a hierarchical planner that decomposes the NP-hard load balancing problem into tractable sub-components, alongside a transfer engine that leverages complementary hardware paths (CPU-assisted and GPU-direct) for overlapped expert transfer. Evaluations on 64 GPUs demonstrate that ForeMoE achieves up to a 1.45× speedup over state-of-the-art RL post-training systems.