mlsanityData to 7 Oct 2026

MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs

arXiv 2411.11217 ↗First posted 18 Nov 2024cs.DC cs.AI cs.LGcited by 73

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Efficient deployment of large language models, particularly Mixture of Experts (MoE), on resource-constrained platforms presents significant challenges, especially in terms of computational efficiency and memory utilization. The MoE architecture, renowned for its ability to increase model capacity without a proportional increase in inference cost, greatly reduces the token generation latency compared with dense models. However, the large model size makes MoE models inaccessible to individuals without high-end GPUs. In this paper, we propose a high-throughput MoE batch inference system, that significantly outperforms past work. MoE-Lightning introduces a novel CPU-GPU-I/O pipelining schedule, CGOPipe, with paged weights to achieve high resource utilization, and a performance model, HRM, based on a Hierarchical Roofline Model we introduce to help find policies with higher throughput than existing systems. MoE-Lightning can achieve up to 10.3x higher throughput than state-of-the-art offloading-enabled LLM inference systems for Mixtral 8x7B on a single T4 GPU (16GB). When the theoretical system throughput is bounded by the GPU memory, MoE-Lightning can reach the throughput upper bound with 2-3x less CPU memory, significantly increasing resource utilization. MoE-Lightning also supports efficient batch inference for much larger MoEs (e.g., Mixtral 8x22B and DBRX) on multiple low-cost GPUs (e.g., 2-4 T4).

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 37 of its 57 references that are in the map. First posted on arXiv 18 Nov 2024.

Builds on

37 of 57 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models10 Feb
  2. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec

Other references · 35

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  9. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  11. 2024Mixtral of Expertscited by 1,785
  12. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
23 more
  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  2. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  3. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  4. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  5. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  6. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  7. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  8. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  9. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  10. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  11. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  12. 2022Efficiently Scaling Transformer Inferencecited by 529
  13. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
  14. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  15. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  16. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  17. 2022Can Foundation Models Wrangle Your Data?cited by 173
  18. 2023Accelerating Transformer Inference for Translation via Parallel Decodingcited by 123
  19. 2020DAPPLE: A Pipelined Data Parallel Approach for Training Large Modelscited by 100
  20. 2024Hydragen: High-Throughput LLM Inference with Shared Prefixescited by 71
  21. 2023Lifelong Language Pretraining with Distribution-Specialized Expertscited by 64
  22. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelinescited by 49
  23. 2021SpreadsheetCoder: Formula Prediction from Semi-structured Contextcited by 41

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 31; 2026: 40.

73 papers.

Sort:

By area:

  1. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  2. 2025MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelismcited by 31
  3. 2025Jenga: Effective Memory Management for Serving LLM with Heterogeneitycited by 24
  4. 2025SLOs-Serve: Optimized Serving of Multi-SLO LLMscited by 17
  5. 2025D2MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Servingcited by 17
  6. 2025L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Modelscited by 14
  7. 2025Utility-Driven Speculative Decoding for Mixture-of-Expertscited by 12
  8. 2025Energy Use of AI Inference, Efficiency Pathways, and Test-Time Scalingcited by 12
  9. 2025eMoE: Task-aware Memory Efficient Mixture-of-Experts-Based (MoE) Model Inferencecited by 11
  10. 2025MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systemscited by 11
  11. 2025FloE: On-the-Fly MoE Inference on Memory-constrained GPUcited by 9
  12. 2025Accelerating Mixture-of-Experts Inference by Hiding Offloading Latency with Speculative Decodingcited by 8
38 more
  1. 2025LayerScope: Predictive Cross-Layer Scheduling for Efficient Multi-Batch MoE Inference on Legacy Serverscited by 8
  2. 2025Janus: Disaggregating Attention and Experts for Scalable MoE Inferencecited by 8
  3. 2026LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructurecited by 8
  4. 2025Patterns behind Chaos: Forecasting Data Movement for Efficient Large-Scale MoE LLM Inferencecited by 6
  5. 2025CHIME: A Case for Efficient Long-Context Attention-FC Disaggregated Inference with DIMM-PIMcited by 5
  6. 2025ExpertFlow: Adaptive Expert Scheduling and Memory Coordination for Efficient MoE Inferencecited by 5
  7. 2026MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarioscited by 5
  8. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inferencecited by 5
  9. 2025A Cost-Effective Near-Storage Processing Solution for Offline Inference of Long-Context LLMscited by 4
  10. 2025BrownoutServe: SLO-Aware Inference Serving under Bursty Workloads for MoE-based LLMscited by 4
  11. 2025FlexiCache: Leveraging Temporal Stability of Attention Heads for Efficient KV Cache Managementcited by 4
  12. 2026ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Schedulingcited by 4
  13. 2026Dynamic Expert Sharing: Decoupling Memory from Parallelism in Mixture-of-Experts Diffusion LLMscited by 4
  14. 2026PCR: A Prefetch-Enhanced Cache Reuse System for Low-Latency RAG Servingcited by 4
  15. 2025Echo: Efficient Co-Scheduling of Hybrid Online-Offline Tasks for Large Language Model Servingcited by 3
  16. 2025MoE-Lens: Towards the Hardware Limit of High-Throughput MoE LLM Serving Under Resource Constraintscited by 3
  17. 2025SMoE: An Algorithm-System Co-Design for Pushing MoE to the Edge via Expert Substitutioncited by 3
  18. 2025ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Cachingcited by 3
  19. 2025Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compressioncited by 3
  20. 2025Bandwidth-Efficient Adaptive Mixture-of-Experts via Low-Rank Compensationcited by 3
  21. 2026An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inferencecited by 3
  22. 2026ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inferencecited by 3
  23. 2025HybridEP: Scaling Expert Parallelism to Cross-Datacenter Scenario via Hybrid Expert/Data Transmissioncited by 2
  24. 2025Context-Aware Mixture-of-Experts Inference on CXL-Enabled GPU-NDP Systemscited by 2
  25. 2026SuperInfer: SLO-Aware Rotary Scheduling and Memory Management for LLM Inference on Superchipscited by 2
  26. 2026Harvest: Opportunistic Peer-to-Peer GPU Caching for LLM Inferencecited by 2
  27. 2026DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCscited by 2
  28. 2026DyMoE: Dynamic Expert Orchestration with Mixed-Precision Quantization for Efficient MoE Inference on Edgecited by 2
  29. 2026HybridGen: Efficient LLM Generative Inference via CPU-GPU Hybrid Computingcited by 2
  30. 2025OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inferencecited by 1
  31. 2025Efficient MoE Inference with Fine-Grained Scheduling of Disaggregated Expert Parallelismcited by 1
  32. 2026MoE-Prefill: Zero Redundancy Overheads in MoE Prefill Servingcited by 1
  33. 2026ShuntServe: Cost-Efficient LLM Serving on Heterogeneous Spot GPU Clusterscited by 1
  34. 2026WiSP: A Working-Set View of Mixture-of-Experts Serving on Extremely Low-Resource Hardwarecited by 1
  35. 2026Beyond Uniform Experts: Cost-Aware Expert Execution for Efficient Multi-Device MoE Inferencecited by 1
  36. 2026FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Executioncited by 1
  37. 2024MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systemscited by 0
  38. 2025APEX: Asynchronous Parallel CPU-GPU Execution for Online LLM Inference on Constrained GPUscited by 0

50 most cited of 73 All 73 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025MoE-Inference-Bench: Performance Evaluation of Mixture of Expert Large Language and Vision Models
  2. 2025MoE-Gen: High-Throughput MoE Inference on a Single GPU with Module-Based Batching
  3. 2025Efficient CPU-GPU Collaborative Inference for MoE-based LLMs on Memory-Limited Systems
  4. 2025Accelerating MoE Model Inference with Expert Sharding
  5. 2026Fast MoE Inference via Predictive Prefetching and Expert Replication
  6. 2024EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inference
  7. 2025SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations
  8. 2025Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism