mlsanityData to 13 Oct 2026

Fast Inference of Mixture-of-Experts Language Models with Offloading

arXiv 2312.17238 ↗First posted 28 Dec 2023cs.LG cs.AI cs.DCcited by 90

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

With the widespread adoption of Large Language Models (LLMs), many deep learning practitioners are looking for strategies of running these models more efficiently. One such strategy is to use sparse Mixture-of-Experts (MoE) - a type of model architectures where only a fraction of model layers are active for any given input. This property allows MoE-based language models to generate tokens faster than their dense counterparts, but it also increases model size due to having multiple experts. Unfortunately, this makes state-of-the-art MoE language models difficult to run without high-end GPUs. In this work, we study the problem of running large MoE language models on consumer hardware with limited accelerator memory. We build upon parameter offloading algorithms and propose a novel strategy that accelerates offloading by taking advantage of innate properties of MoE LLMs. Using this strategy, we build can run Mixtral-8x7B with mixed quantization on desktop hardware and free-tier Google Colab instances.

Key facts. Cited by 90 papers in the map (data through 13 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 31 of its 39 references that are in the map. First posted on arXiv 28 Dec 2023.

Builds on

31 of 39 references.

References · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,876
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,317
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,500
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,463
  8. 2016Pointer Sentinel Mixture Modelscited by 3,047
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,044
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,978
  11. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,884
  12. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,761
19 more
  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,618
  2. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,272
  3. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,103
  4. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 896
  5. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 896
  6. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 814
  7. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 617
  8. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 570
  9. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 406
  10. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 365
  11. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 340
  12. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  13. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 263
  14. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 249
  15. 2022Language model compression with weighted low-rank factorizationcited by 192
  16. 2019Large Memory Layers with Product Keyscited by 121
  17. 2023QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Modelscited by 42
  18. 2020Training Large Neural Networks with Constant Memory using a New Execution Algorithmcited by 34
  19. 2023Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustnesscited by 34

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 20; 2025: 34; 2026: 36.

90 papers.

Sort:

By area:

  1. 2024Mixtral of Expertscited by 1,811
  2. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73
  3. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Modelscited by 69
  4. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inferencecited by 54
  5. 2024MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cachecited by 49
  6. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  7. 2024Harnessing the Power of Multiple Minds: Lessons Learned from LLM Routingcited by 45
  8. 2024AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inferencecited by 42
  9. 2024ProMoE: Fast MoE-based LLM Serving using Proactive Cachingcited by 39
  10. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloadingcited by 24
  11. 2025Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipelinecited by 21
  12. 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Schedulingcited by 19
38 more
  1. 2024Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Designcited by 18
  2. 2024Lynx: Enabling Efficient MoE Inference through Dynamic Batch-Aware Expert Selectioncited by 16
  3. 2025HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inferencecited by 16
  4. 2024Mixture of Cache-Conditional Experts for Efficient Mobile Device Inferencecited by 14
  5. 2024DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inferencecited by 14
  6. 2025Fate: Fast Edge Inference of Mixture-of-Experts Models via Cross-Layer Gatecited by 13
  7. 2025GRACE-MoE: Grouping and Replication with Locality-Aware Routing for Efficient Distributed MoE Inferencecited by 13
  8. 2024Efficient Heterogeneous Large Language Model Decoding with Model-Attention Disaggregationcited by 12
  9. 2025Mixture of Lookup Expertscited by 12
  10. 2025SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inferencecited by 12
  11. 2025MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systemscited by 11
  12. 2025MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading for Mixture-of-Expertscited by 11
  13. 2025FloE: On-the-Fly MoE Inference on Memory-constrained GPUcited by 9
  14. 2025Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanismcited by 9
  15. 2024Revisiting SMoE Language Models by Evaluating Inefficiencies with Task Specific Expert Pruningcited by 8
  16. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
  17. 2024Mixture of Experts with Mixture of Precisions for Tuning Quality of Servicecited by 7
  18. 2025MiLo: Efficient Quantized MoE Inference with Mixture of Low-Rank Compensatorscited by 7
  19. 2025MoE-Gen: High-Throughput MoE Inference on a Single GPU with Module-Based Batchingcited by 6
  20. 2025Patterns behind Chaos: Forecasting Data Movement for Efficient Large-Scale MoE LLM Inferencecited by 6
  21. 2025Dynamic Expert Quantization for Scalable Mixture-of-Experts Inferencecited by 6
  22. 2025CoServe: Efficient Collaboration-of-Experts (CoE) Model Inference with Limited Memorycited by 5
  23. 2025ZO2: Scalable Zeroth-Order Fine-Tuning for Extremely Large Language Models with Limited GPU Memorycited by 5
  24. 2025ExpertFlow: Adaptive Expert Scheduling and Memory Coordination for Efficient MoE Inferencecited by 5
  25. 2026MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarioscited by 5
  26. 2024Achieving Peak Performance for Large Language Models: A Systematic Reviewcited by 4
  27. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  28. 2026ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Schedulingcited by 4
  29. 2026SpecMoE: A Fast and Efficient Mixture-of-Experts Inference via Self-Assisted Speculative Decodingcited by 4
  30. 2025CoSMoEs: Compact Sparse Mixture of Expertscited by 3
  31. 2025MoE-Lens: Towards the Hardware Limit of High-Throughput MoE LLM Serving Under Resource Constraintscited by 3
  32. 2025Toward Cost-Efficient Serving of Mixture-of-Experts with Asynchronycited by 3
  33. 2025On-the-Fly Adaptive Distillation of Transformer to Dual-State Linear Attentioncited by 3
  34. 2025Bandwidth-Efficient Adaptive Mixture-of-Experts via Low-Rank Compensationcited by 3
  35. 2026Temporally Extended Mixture-of-Experts Modelscited by 3
  36. 2024Model Agnostic Hybrid Sharding For Heterogeneous Distributed Inferencecited by 2
  37. 2025RAGDoll: Efficient Offloading-based Online RAG System on a Single GPUcited by 2
  38. 2025In-depth Analysis on Caching and Pre-fetching in Mixture of Experts Offloadingcited by 2

50 most cited of 90 All 90 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
  2. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference
  3. 2025MoE-Inference-Bench: Performance Evaluation of Mixture of Expert Large Language and Vision Models
  4. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale
  5. 2021Scalable and Efficient MoE Training for Multitask Multilingual Models
  6. 2025Faster MoE LLM Inference for Extremely Large Models
  7. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
  8. 2024MoNDE: Mixture of Near-Data Experts for Large-Scale Sparse Models