mlsanityData to 14 Oct 2026

Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models

arXiv 2607.01844 ↗First posted 2 Jul 2026cs.DC cs.AIcited by 1

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

This paper showcases a memory-efficient training stack for Mixture-of-Experts (MoE) models. It is a training paradigm that combines and specializes various existing and novel parallelism techniques at different layers and stages of the Mixture-of-Experts (MoE) model training pipeline. It leverages these techniques to achieve maximal efficiency given the physical constraints of CPU, CPU memory, GPU HBM memory, and the CPU-GPU, GPU-GPU, and node-node communication bandwidth of the GPU cluster. It also contains a novel strategy for the optimizer step to achieve high throughput and memory efficiency, enabling practitioners to conduct lossless pre-training/fine-tuning of trillion-parameter scale models, at a million context length, with just under 12 8x H200 GPU nodes, with state-of-the-art throughput and memory efficiency. In our experiments, MoP delivers 4.7x--8.2x higher per-GPU throughput than a strongly-tuned FSDP2 baseline (with the gap widening at larger scale) and sustains training at context lengths up to 1M tokens, where the baseline runs out of memory beyond 64--128K.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 13 of its 14 references that are in the map. First posted on arXiv 2 Jul 2026.

Builds on

13 of 14 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-Experts23 Jan
  2. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 11

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  3. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  5. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  6. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  7. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  8. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  9. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  10. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 245
  11. 2024TorchTitan: One-stop PyTorch native solution for production ready LLM pre-trainingcited by 85

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Flattening Every Memory Peak in Long-Context Mixture-of-Experts Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Scalable Training of Mixture-of-Experts Models with Megatron Core
  2. 2026MoE-DisCo:Low Economy Cost Training Mixture-of-Experts Models
  3. 2023A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training
  4. 2026MELINOE: Fine-Tuning Enables Memory-Efficient Inference for Mixture-of-Experts Models
  5. 2024Communication-Efficient Sparsely-Activated Model Training via Sequence Migration and Token Condensation
  6. 2025HAP: Hybrid Adaptive Parallelism for Efficient Mixture-of-Experts Inference
  7. 2025X-MoE: Enabling Scalable Training for Emerging Mixture-of-Experts Architectures on HPC Platforms
  8. 2025MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production