mlsanityData to 14 Oct 2026

Hexa-MoE: Efficient and Heterogeneous-aware Training for Mixture-of-Experts

arXiv 2411.01288 ↗First posted 2 Nov 2024cs.DCcited by 1

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture-of-Experts (MoE) has emerged as a practical approach to scale up parameters for the Transformer model to achieve better generalization while maintaining a sub-linear increase in computation overhead. Current MoE models are mainly built with expert parallelism on distributed devices. However, it usually depends on homogeneous devices to deploy and suffers from heavy communication overhead and computation redundancy. In this paper, we explore developing a textttHeterogeneous-aware textttEXpert textttAllocation framework, textbftextttHEXA-MoE, with significantly enhanced computing efficiency. It contains two components: (1) textitExpert-Specific Operators. We replace the typical general matrix multiplication or grouped matrix multiplication interfaces with our operators, which allows the computing to be performed in an in-place manner with ZERO redundancy. (2) textitAdaptive Data- and Model-Centric Configurations for different workload scales. Specifically, we introduce a pipeline-shared cache on each device to tackle the heavy memory consumption in the existing data-centric MoE library. Comprehensive experiments on the Swin-MoE benchmark consistently reveal the effectiveness of our textttHEXA-MoE framework, i.e., reducing 10%~48% memory consumption and achieving 0.5~4.3× speed up compared to current state-of-the-art MoE libraries. Furthermore, we examine our textttHEXA-MoE with heterogeneous devices for both data- and model-centric settings. Promising results show that employing optimal parallel configuration with textttHEXA-MoE on heterogeneous devices can substantially minimize overall latency. Codes are available at https://github.com/UNITES-Lab/HEXA-MoE.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 18 of its 26 references that are in the map. First posted on arXiv 2 Nov 2024.

Builds on

18 of 26 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts29 Nov
  2. 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct
  3. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun

Other references · 15

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  9. 2024Mixtral of Expertscited by 1,823
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  11. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  12. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
3 more
  1. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  2. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
  3. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 98

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025SYMI: Efficient Mixture-of-Experts Training via Model and Optimizer State Decouplingcited by 5

Similar, not cited

Close in wording, no citation either way

  1. 2025Union of Experts: Adapting Hierarchical Routing to Equivalently Decomposed Transformer
  2. 2026LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Training
  3. 2025Optimal Scaling Laws for Efficiency Gains in a Theoretical Transformer-Augmented Sectional MoE Framework
  4. 2024Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts
  5. 2026Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models
  6. 2025MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing
  7. 2026DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices
  8. 2026Scalable Training of Mixture-of-Experts Models with Megatron Core