mlsanityData to 14 Oct 2026

Mixture of Experts with Mixture of Precisions for Tuning Quality of Service

arXiv 2407.14417 ↗First posted 19 Jul 2024cs.DC cs.AI cs.LGcited by 7

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

The increasing demand for deploying large Mixture-of-Experts (MoE) models in resource-constrained environments necessitates efficient approaches to address their high memory and computational requirements challenges. Moreover, given that tasks come in different user-defined constraints and the available resources change over time in multi-tenant environments, it is necessary to design an approach which provides a flexible configuration space. This paper presents an adaptive serving approach for the efficient deployment of MoE models, capitalizing on partial quantization of the experts. By dynamically determining the number of quantized experts and their distribution across CPU and GPU, our approach explores the Pareto frontier and offers a fine-grained range of configurations for tuning throughput and model quality. Our evaluation on an NVIDIA A100 GPU using a Mixtral 8x7B MoE model for three language modelling benchmarks demonstrates that the throughput of token generation can be adjusted from 0.63 to 13.00 token per second. This enhancement comes with a marginal perplexity increase of 3.81 to 4.00, 13.59 to 14.17, and 7.24 to 7.40 for WikiText2, PTB, and C4 datasets respectively under maximum quantization. These results highlight the practical applicability of our approach in dynamic and accuracy-sensitive applications where both memory usage and output quality are important.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 25 of its 33 references that are in the map. First posted on arXiv 19 Jul 2024.

Builds on

25 of 33 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models10 Feb
  2. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec
  3. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 22

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  8. 2024Mixtral of Expertscited by 1,823
  9. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  10. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  11. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  12. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
10 more
  1. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  2. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 169
  3. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 144
  4. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 113
  5. 2021Scalable and Efficient MoE Training for Multitask Multilingual Modelscited by 76
  6. 2023QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Modelscited by 42
  7. 2023Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustnesscited by 34
  8. 2021M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretrainingcited by 31
  9. 2022Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Productioncited by 27
  10. 2023Sparse Backpropagation for MoE Trainingcited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 2.

7 papers.

  1. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 47
  2. 2025MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoEcited by 19
  3. 2025EAQuant: Enhancing Post-Training Quantization for MoE Models via Expert-Aware Optimizationcited by 10
  4. 2026MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarioscited by 5
  5. 2025Dual Mutual Learning Network with Global-local Awareness for RGB-D Salient Object Detectioncited by 0
  6. 2025QoS-Efficient Serving of Multiple Mixture-of-Expert LLMs Using Partial Runtime Reconfigurationcited by 0
  7. 2026Every Expert Counts: ExactMoE for Memory-Efficient W4A16 Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Mixture of Diverse Size Experts
  2. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs
  3. 2026Mixture of Heterogeneous Grouped Experts for Language Modeling
  4. 2026Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Experts
  5. 2024Optimizing Mixture-of-Experts Inference Time Combining Model Deployment and Communication Scheduling
  6. 2025Dynamic Expert Quantization for Scalable Mixture-of-Experts Inference
  7. 2026Moebius: Serving Mixture-of-Expert Models with Seamless Runtime Parallelism Switch
  8. 2025MoE-Inference-Bench: Performance Evaluation of Mixture of Expert Large Language and Vision Models