mlsanityData to 14 Oct 2026

Accelerating Distributed MoE Training and Inference with Lina

arXiv 2210.17223 ↗First posted 31 Oct 2022cs.DCcited by 109

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Scaling model parameters improves model quality at the price of high computation overhead. Sparsely activated models, usually in the form of Mixture of Experts (MoE) architecture, have sub-linear scaling of computation cost with model size, thus providing opportunities to train and serve a larger model at lower cost than their dense counterparts. However, distributed MoE training and inference is inefficient, mainly due to the interleaved all-to-all communication during model computation. This paper makes two main contributions. First, we systematically analyze all-to-all overhead in distributed MoE and present the main causes for it to be the bottleneck in training and inference, respectively. Second, we design and build Lina to address the all-to-all bottleneck head-on. Lina opportunistically prioritizes all-to-all over the concurrent allreduce whenever feasible using tensor partitioning, so all-to-all and training step time is improved. Lina further exploits the inherent pattern of expert selection to dynamically schedule resources during inference, so that the transfer size and bandwidth of all-to-all across devices are balanced amid the highly skewed expert popularity in practice. Experiments on an A100 GPU testbed show that Lina reduces the training step time by up to 1.73x and reduces the 95%ile inference time by an average of 1.63x over the state-of-the-art systems.

Key facts. Cited by 109 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 28 of its 49 references that are in the map. First posted on arXiv 31 Oct 2022.

Builds on

28 of 49 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Services20 May
  2. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  8. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  11. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  12. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
14 more
  1. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  2. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  3. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  4. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 179
  5. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  6. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 169
  7. 2015Conditional Computation in Neural Networks for faster modelscited by 167
  8. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 157
  9. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 113
  10. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 98
  11. 2022Pathways: Asynchronous Distributed Dataflow for MLcited by 82
  12. 2017Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusterscited by 57
  13. 2019Blink: Fast and Generic Collectives for Distributed MLcited by 36
  14. 2018TicTac: Accelerating Distributed Deep Learning with Communication Schedulingcited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 17; 2025: 49; 2026: 38.

109 papers.

Sort:

By area:

  1. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  2. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 193
  3. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 163
  4. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inferencecited by 104
  5. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 102
  6. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 72
  7. 2025Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decodingcited by 55
  8. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 47
  9. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  10. 2025MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelismcited by 32
  11. 2024LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scalecited by 31
  12. 2025MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Productioncited by 31
38 more
  1. 2024Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inferencecited by 28
  2. 2025MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routingcited by 26
  3. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloadingcited by 24
  4. 2024Lancet: Accelerating Mixture-of-Experts Training via Whole Graph Computation-Communication Overlappingcited by 22
  5. 2025Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipelinecited by 21
  6. 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Schedulingcited by 19
  7. 2023From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscapecited by 18
  8. 2025FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Modelscited by 17
  9. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  10. 2025Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantizationcited by 14
  11. 2024Toward Sustainable GenAI using Generation Directives for Carbon-Friendly Large Language Model Inferencecited by 13
  12. 2025Fate: Fast Edge Inference of Mixture-of-Experts Models via Cross-Layer Gatecited by 13
  13. 2025GRACE-MoE: Grouping and Replication with Locality-Aware Routing for Efficient Distributed MoE Inferencecited by 13
  14. 2024Training and Serving System of Foundation Models: A Comprehensive Surveycited by 11
  15. 2025eMoE: Task-aware Memory Efficient Mixture-of-Experts-Based (MoE) Model Inferencecited by 11
  16. 2025MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systemscited by 11
  17. 2025TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Schedulingcited by 10
  18. 2025Stratum: System-Hardware Co-Design with Tiered Monolithic 3D-Stackable DRAM for Efficient MoE Servingcited by 10
  19. 2025Semantic Parallelism: Redefining Efficient MoE Inference via Model-Data Co-Schedulingcited by 9
  20. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
  21. 2025LayerScope: Predictive Cross-Layer Scheduling for Efficient Multi-Batch MoE Inference on Legacy Serverscited by 8
  22. 2025Janus: Disaggregating Attention and Experts for Scalable MoE Inferencecited by 8
  23. 2026Mixture of Universal Experts: Scaling Virtual Width via Depth-Width Transformationcited by 8
  24. 2024Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedulescited by 7
  25. 2024Optimizing Mixture-of-Experts Inference Time Combining Model Deployment and Communication Schedulingcited by 7
  26. 2025InfiniteHBD: Building Datacenter-Scale High-Bandwidth Domain for LLM with Optical Circuit Switching Transceiverscited by 7
  27. 2025Progressive Sparse Attention: Algorithm and System Co-design for Efficient Attention in LLM Servingcited by 7
  28. 2025HeterMoE: Efficient Training of Mixture-of-Experts Models on Heterogeneous GPUscited by 7
  29. 2025Lumos: Efficient Performance Modeling and Estimation for Large-scale LLM Trainingcited by 7
  30. 2025HarMoEny: Efficient Multi-GPU Inference of MoE Modelscited by 7
  31. 2025Accelerating Edge Inference for Distributed MoE Models with Latency-Optimized Expert Placementcited by 7
  32. 2025MoE-Gen: High-Throughput MoE Inference on a Single GPU with Module-Based Batchingcited by 6
  33. 2025SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Servingcited by 6
  34. 2025Patterns behind Chaos: Forecasting Data Movement for Efficient Large-Scale MoE LLM Inferencecited by 6
  35. 2026LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Trainingcited by 6
  36. 2025SYMI: Efficient Mixture-of-Experts Training via Model and Optimizer State Decouplingcited by 5
  37. 2025TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Servingcited by 5
  38. 2025DuoServe-MoE: Dual-Phase Expert Prefetch and Caching for LLM Inference QoS Assurancecited by 5

50 most cited of 109 All 109 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Communication-Efficient Sparsely-Activated Model Training via Sequence Migration and Token Condensation
  2. 2026Fast MoE Inference via Predictive Prefetching and Expert Replication
  3. 2025OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference
  4. 2025LExI: Layer-Adaptive Active Experts for Efficient MoE Model Inference
  5. 2026EMO: Frustratingly Easy Progressive Training of Extendable MoE
  6. 2025Dynamic Expert Quantization for Scalable Mixture-of-Experts Inference
  7. 2026MELINOE: Fine-Tuning Enables Memory-Efficient Inference for Mixture-of-Experts Models
  8. 2026Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models