mlsanityData to 14 Oct 2026

ReLibra: Routing-Replay-Guided Load Balancing for MoE Training in Reinforcement Learning

arXiv 2605.08639 ↗First posted 9 May 2026cs.LGcited by 3

On the map: Mixture-of-experts → MoE in reinforcement learning and robotics

also in 1 more
Abstract (arXiv)

Load imbalance is a long-standing challenge in Mixture-of-Experts (MoE) training and is exacerbated in reinforcement learning (RL) for LLMs, where hot experts can shift frequently across micro-batches. Existing MoE training systems rely on historical loads to predict future expert demand, making them less effective under sharp fluctuations. We propose ReLibra, an MoE RL training system that exploits a unique opportunity in RL's rollout-training workflow, routing replay, to enable fine-grained load balancing at micro-batch granularity. Because rollout and training process the same tokens with the same MoE parameters, the token-to-expert routing decisions are known before training starts. Leveraging this information, ReLibra places two MoE load-balancing mechanisms at inter- and intra-batch timescales, matching their communication patterns to hierarchical network bandwidths. At the inter-batch timescale, ReLibra performs expert reordering to redistribute experts for batch-level cross-node balancing; at the intra-batch timescale, it dynamically performs expert replication within a node to absorb micro-batch-level load fluctuations. Experiments on diverse MoE LLMs and RL workloads show that ReLibra improves training throughput by up to 1.6× over Megatron-LM and by up to 1.2× over EPLB, even when EPLB is given oracle loads. Moreover, ReLibra remains within 6%-10% of the throughput of an idealized balanced baseline.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in MoE in reinforcement learning and robotics, one of 12 directions in Mixture-of-experts. Builds on 50 of its 78 references that are in the map. First posted on arXiv 9 May 2026.

Builds on

50 of 78 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practices1 Dec
  2. 2025Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers13 Oct
  3. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models8 Aug
  4. 2025Kimi K2: Open Agentic Intelligence28 Jul
  5. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention16 Jun

Other references · 45

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  9. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  10. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  11. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  12. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
33 more
  1. 2024Mixtral of Expertscited by 1,823
  2. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  4. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  5. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  6. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  7. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  8. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  9. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
  10. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  11. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
  12. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  13. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  14. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 201
  15. 2025Generalizing Verifiable Instruction Followingcited by 194
  16. 2023Fast Distributed Inference Serving for Large Language Modelscited by 145
  17. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 112
  18. 2022Accelerating Distributed MoE Training and Inference with Linacited by 109
  19. 2025StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generationcited by 61
  20. 2024FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusioncited by 57
  21. 2023FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placementcited by 53
  22. 2025Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Expertscited by 49
  23. 2024Optimizing RLHF Training for Large Language Models with Stage Fusioncited by 43
  24. 2024The SCIP Optimization Suite 9.0cited by 37
  25. 2022HetuMoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training Systemcited by 36
  26. 2025MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Productioncited by 31
  27. 2025Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learningcited by 25
  28. 2025MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Corecited by 17
  29. 2024DistTrain: Addressing Model and Data Heterogeneity with Disaggregated Training for Multimodal Large Language Modelscited by 16
  30. 2025Taming the Long-Tail: Efficient Reasoning RL Training with Adaptive Draftercited by 14
  31. 2025RollArt: Disaggregated Multi-Task Agentic RL Training at Scalecited by 14
  32. 2026Heddle: A Distributed Orchestration System for Agentic RL Rolloutcited by 8
  33. 2026LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Trainingcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Harnessing Routing Foresight for Micro-step-level MoE load balancing in RL Post-trainingcited by 2
  2. 2026RoutePack: Expert Placement and Attention-Aware Data Packing for MoE Reinforcement Learningcited by 0
  3. 2026CIPHER-MoE: Balancing Efficiency and Routing Fidelity in Trillion-Scale MoE Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Stable and Effective Reinforcement Learning for Mixture-of-Experts
  2. 2026Three Phases of Expert Routing: How Load Balance Evolves During Mixture-of-Experts Training
  3. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts
  4. 2026A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMs
  5. 2026PR2: Predictive Routing Replay for MoE-Based LLM Reinforcement Learning
  6. 2026Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-Experts
  7. 2026Phase-Aware Mixture of Experts for Agentic Reinforcement Learning
  8. 2026φ-Balancing for Mixture-of-Experts Training