mlsanityData to 14 Oct 2026

MemFine: Memory-Aware Fine-Grained Scheduling for MoE Training

arXiv 2511.21431 ↗First posted 26 Nov 2025cs.DCcited by 1

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

The training of large-scale Mixture of Experts (MoE) models faces a critical memory bottleneck due to severe load imbalance caused by dynamic token routing. This imbalance leads to memory overflow on GPUs with limited capacity, constraining model scalability. Existing load balancing methods, which cap expert capacity, compromise model accuracy and fail on memory-constrained hardware. To address this, we propose MemFine, a memory-aware fine-grained scheduling framework for MoE training. MemFine decomposes the token distribution and expert computation into manageable chunks and employs a chunked recomputation strategy, dynamically optimized through a theoretical memory model to balance memory efficiency and throughput. Experiments demonstrate that MemFine reduces activation memory by 48.03% and improves throughput by 4.42% compared to full recomputation-based baselines, enabling stable large-scale MoE training on memory-limited GPUs.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 18 of its 23 references that are in the map. First posted on arXiv 26 Nov 2025.

Builds on

18 of 23 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding25 Jul
  2. 2025MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production16 May
  3. 2025Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUs7 May
  4. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 14

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  6. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  7. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  8. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  9. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  10. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  11. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 205
  12. 2022Deep CSI Compression for Massive MIMO: A Self-information Model-driven Neural Networkcited by 9
2 more
  1. 2022Data Augmentation Empowered Neural Precoding for Multiuser MIMO with MMSE Modelcited by 4
  2. 2025MoFa: A Unified Performance Modeling Framework for LLM Pretrainingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Memory-Efficient Expert Routing for Distributed MoE Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MoEBlaze: Breaking the Memory Wall for Efficient MoE Training on Modern GPUs
  2. 2025Fine-grained MoE Load Balancing with Linear Programming
  3. 2026Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models
  4. 2026Scalable Training of Mixture-of-Experts Models with Megatron Core
  5. 2025ExpertFlow: Adaptive Expert Scheduling and Memory Coordination for Efficient MoE Inference
  6. 2025Accelerating MoE Model Inference with Expert Sharding
  7. 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling
  8. 2025Efficient MoE Serving in the Memory-Bound Regime: Balance Activated Experts, Not Tokens