# MemFine: Memory-Aware Fine-Grained Scheduling for MoE Training

arXiv: https://arxiv.org/abs/2511.21431 · HTML: https://mlsanity.com/paper/2511.21431/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts.
- Builds on 18 of its 23 references that are in the map.
- First posted on arXiv 26 Nov 2025.

## Placed in

- Mixture-of-experts / [Efficient MoE training and inference systems](https://mlsanity.com/moe/efficient-moe-training-and-inference-systems/)

## Builds on (most cited first, 18 of 18)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale](https://mlsanity.com/paper/2201.05596/) (2022)
- [ZeRO-Offload: Democratizing Billion-Scale Model Training](https://mlsanity.com/paper/2101.06840/) (2021)
- [Reducing Activation Recomputation in Large Transformer Models](https://mlsanity.com/paper/2205.05198/) (2022)
- [A Survey on Mixture of Experts in Large Language Models](https://mlsanity.com/paper/2407.06204/) (2024)
- [Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts](https://mlsanity.com/paper/2408.15664/) (2024)
- [Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding](https://mlsanity.com/paper/2507.19427/) (2025)
- [MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production](https://mlsanity.com/paper/2505.11432/) (2025)
- [Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUs](https://mlsanity.com/paper/2505.04519/) (2025)
- [Deep CSI Compression for Massive MIMO: A Self-information Model-driven Neural Network](https://mlsanity.com/paper/2204.11567/) (2022)
- [Data Augmentation Empowered Neural Precoding for Multiuser MIMO with MMSE Model](https://mlsanity.com/paper/2203.02196/) (2022)
- [MoFa: A Unified Performance Modeling Framework for LLM Pretraining](https://mlsanity.com/paper/2511.09837/) (2025)

## Built on by (most cited first, 1 of 1)

- [Memory-Efficient Expert Routing for Distributed MoE Training](https://mlsanity.com/paper/2610.07333/) (2026)

## Abstract (arXiv)

The training of large-scale Mixture of Experts (MoE) models faces a critical memory bottleneck due to severe load imbalance caused by dynamic token routing. This imbalance leads to memory overflow on GPUs with limited capacity, constraining model scalability. Existing load balancing methods, which cap expert capacity, compromise model accuracy and fail on memory-constrained hardware. To address this, we propose MemFine, a memory-aware fine-grained scheduling framework for MoE training. MemFine decomposes the token distribution and expert computation into manageable chunks and employs a chunked recomputation strategy, dynamically optimized through a theoretical memory model to balance memory efficiency and throughput. Experiments demonstrate that MemFine reduces activation memory by 48.03% and improves throughput by 4.42% compared to full recomputation-based baselines, enabling stable large-scale MoE training on memory-limited GPUs.