mlsanityData to 14 Oct 2026

MoEBlaze: Breaking the Memory Wall for Efficient MoE Training on Modern GPUs

arXiv 2601.05296 ↗First posted 8 Jan 2026cs.LG cs.AI cs.DCcited by 2

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

The pervasive "memory wall" bottleneck is significantly amplified in modern large-scale Mixture-of-Experts (MoE) architectures. MoE's inherent architectural sparsity leads to sparse arithmetic compute and also introduces substantial activation memory overheads -- driven by large token routing buffers and the need to materialize and buffer intermediate tensors. This memory pressure limits the maximum batch size and sequence length that can fit on GPUs, and also results in excessive data movements that hinders performance and efficient model scaling. We present MoEBlaze, a memory-efficient MoE training framework that addresses these issues through a co-designed system approach: (i) an end-to-end token dispatch and MoE training method with optimized data structures to eliminate intermediate buffers and activation materializing, and (ii) co-designed kernels with smart activation checkpoint to mitigate memory footprint while simultaneously achieving better performance. We demonstrate that MoEBlaze can achieve over 4x speedups and over 50% memory savings compared to existing MoE frameworks.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 15 of its 23 references that are in the map. First posted on arXiv 8 Jan 2026.

Builds on

15 of 23 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts29 Nov
  2. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  3. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan
  4. 2021FastMoE: A Fast Mixture-of-Expert Training System24 Mar

Other references · 11

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  6. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  7. 2024Mixtral of Expertscited by 1,823
  8. 2020GLU Variants Improve Transformercited by 1,804
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  10. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  11. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026UniEP: Unified Expert-Parallel MoE MegaKernel for LLM Trainingcited by 5
  2. 2026Memory-Efficient Expert Routing for Distributed MoE Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MemFine: Memory-Aware Fine-Grained Scheduling for MoE Training
  2. 2025SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations
  3. 2025X-MoE: Enabling Scalable Training for Emerging Mixture-of-Experts Architectures on HPC Platforms
  4. 2024Communication-Efficient Sparsely-Activated Model Training via Sequence Migration and Token Condensation
  5. 2025Accelerating MoE Model Inference with Expert Sharding
  6. 2025Efficient MoE Serving in the Memory-Bound Regime: Balance Activated Experts, Not Tokens
  7. 2026A Scheduling Framework for Efficient MoE Inference on Edge GPU-NDP Systems
  8. 2025MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing