mlsanityData to 14 Oct 2026

HarMoEny: Efficient Multi-GPU Inference of MoE Models

arXiv 2506.12417 ↗First posted 14 Jun 2025cs.DCcited by 7

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture-of-Experts (MoE) models offer computational efficiency during inference by activating only a subset of specialized experts for a given input. This enables efficient model scaling on multi-GPU systems that use expert parallelism without compromising performance. However, load imbalance among experts and GPUs introduces waiting times, which can significantly increase inference latency. To address this challenge, we propose HarMoEny, a novel solution to address MoE load imbalance through two simple techniques: (i) dynamic token redistribution to underutilized GPUs and (ii) asynchronous prefetching of experts from the system to GPU memory. These techniques achieve a near-perfect load balance among experts and GPUs and mitigate delays caused by overloaded GPUs. We implement HarMoEny and compare its latency and throughput with four MoE baselines using real-world and synthetic datasets. Under heavy load imbalance, HarMoEny increases throughput by 37%-70% and reduces time-to-first-token by 34%-41%, compared to the next-best baseline. Moreover, our ablation study demonstrates that HarMoEny's scheduling policy reduces the GPU idling time by up to 84% compared to the baseline policies.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 19 of its 38 references that are in the map. First posted on arXiv 14 Jun 2025.

Builds on

19 of 38 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference16 Jan
  2. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts29 Nov
  3. 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct
  4. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  5. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan
  6. 2021FastMoE: A Fast Mixture-of-Expert Training System24 Mar

Other references · 13

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  8. 2016Pointer Sentinel Mixture Modelscited by 3,096
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  11. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  12. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
1 more
  1. 2019Character-based NMT with Transformercited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026PROBE: Co-Balancing Computation and Communication in MoE Inference via Real-Time Predictive Prefetchingcited by 4
  2. 2026Harvest: Opportunistic Peer-to-Peer GPU Caching for LLM Inferencecited by 2
  3. 2026MELINOE: Fine-Tuning Enables Memory-Efficient Inference for Mixture-of-Experts Modelscited by 2
  4. 2026A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMscited by 0
  5. 2026Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Schedulingcited by 0
  6. 2026GEM: GPU-Variability-Aware Expert to GPU Mapping for MoE Systemscited by 0
  7. 2026EasyBalance: Cross-Layer Load Balancing in Distributed MoE Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026TriMoE: Augmenting GPU with AMX-Enabled CPU and DIMM-NDP for High-Throughput MoE Inference via Offloading
  2. 2026A Scheduling Framework for Efficient MoE Inference on Edge GPU-NDP Systems
  3. 2025Accelerating MoE Model Inference with Expert Sharding
  4. 2024DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inference
  5. 2025Efficient MoE Inference with Fine-Grained Scheduling of Disaggregated Expert Parallelism
  6. 2025MoBiLE: Efficient Mixture-of-Experts Inference on Consumer GPU with Mixture of Big Little Experts
  7. 2024Optimizing Mixture-of-Experts Inference Time Combining Model Deployment and Communication Scheduling
  8. 2025FloE: On-the-Fly MoE Inference on Memory-constrained GPU