mlsanityData to 14 Oct 2026

Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Production

arXiv 2211.10017 ↗First posted 18 Nov 2022cs.CL cs.AI cs.LGcited by 27

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Mixture of Experts (MoE) models with conditional execution of sparsely activated layers have enabled training models with a much larger number of parameters. As a result, these models have achieved significantly better quality on various natural language processing tasks including machine translation. However, it remains challenging to deploy such models in real-life scenarios due to the large memory requirements and inefficient inference. In this work, we introduce a highly efficient inference framework with several optimization approaches to accelerate the computation of sparse models and cut down the memory consumption significantly. While we achieve up to 26x speed-up in terms of throughput, we also reduce the model size almost to one eighth of the original 32-bit float model by quantizing expert weights into 4-bit integers. As a result, we are able to deploy 136x larger models with 27% less cost and significantly better quality compared to the existing solutions. This enables a paradigm shift in deploying large scale multilingual MoE transformers models replacing the traditional practice of distilling teacher models into dozens of smaller models per language or task.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 12 of its 16 references that are in the map. First posted on arXiv 18 Nov 2022.

Builds on

12 of 16 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformers28 May
  2. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  3. 2021Scalable and Efficient MoE Training for Multitask Multilingual Models22 Sep
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 7

  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  2. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  3. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  4. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  5. 2020Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationcited by 188
  6. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 169
  7. 2020Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translationcited by 105

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 9; 2025: 10; 2026: 4.

27 papers.

Sort:

By area:

  1. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  2. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  3. 2024MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Modelscited by 52
  4. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 49
  5. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 47
  6. 2025TileLang: A Composable Tiled Programming Model for AI Systemscited by 43
  7. 2023QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Modelscited by 42
  8. 2024A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithmscited by 31
  9. 2025MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Designcited by 30
  10. 2023FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMscited by 22
  11. 2023FPTQ: Fine-grained Post-Training Quantization for Large Language Modelscited by 16
  12. 2024QQQ: Quality Quattuor-Bit Quantization for Large Language Modelscited by 16
15 more
  1. 2024Scattered Mixture-of-Experts Implementationcited by 15
  2. 2025Ecco: Improving Memory Bandwidth and Capacity for LLMs via Entropy-aware Cache Compressioncited by 14
  3. 2024Speed of Light Exact Greedy Decoding for RNN-T Speech Recognition Models on GPUcited by 12
  4. 2025LMDeploy Accelerates Mixed-Precision LLM Inference with TurboMindcited by 10
  5. 2025FIGLUT: An Energy-Efficient Accelerator Design for FP-INT GEMM Using Look-Up Tablescited by 9
  6. 2025BitDecoding: Unlocking Tensor Cores for Long-Context LLMs with Low-Bit KV Cachecited by 8
  7. 2025MiLo: Efficient Quantized MoE Inference with Mixture of Low-Rank Compensatorscited by 8
  8. 2024Mixture of Experts with Mixture of Precisions for Tuning Quality of Servicecited by 7
  9. 2024EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inferencecited by 6
  10. 2026MatGPTQ: Efficient and Accurate Inference over Nested Quantized Modelscited by 3
  11. 2026Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guaranteescited by 2
  12. 2025HipKittens: Fast and Furious AMD Kernelscited by 1
  13. 2026Skill Weaving: Efficient LLM Improvement via Modular Skillpackscited by 1
  14. 2025PacQ: A SIMT Microarchitecture for Efficient Dataflow in Hyper-asymmetric GEMMscited by 0
  15. 2026Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Stagingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Faster MoE LLM Inference for Extremely Large Models
  2. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
  3. 2023Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inference
  4. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference
  5. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading
  6. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
  7. 2025Mixture of Experts in Large Language Models
  8. 2025Efficiently Editing Mixture-of-Experts Models with Compressed Experts