mlsanityData to 7 Oct 2026

Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness

arXiv 2310.02410 ↗First posted 3 Oct 2023cs.LG cs.CLcited by 34

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Large Mixture of Experts (MoE) models could achieve state-of-the-art quality on various language tasks, including machine translation task, thanks to the efficient model scaling capability with expert parallelism. However, it has brought a fundamental issue of larger memory consumption and increased memory bandwidth bottleneck at deployment time. In this paper, we propose Mixture of Quantized Experts (MoQE) which is a simple weight-only quantization method applying ultra low-bit down to 2-bit quantizations only to expert weights for mitigating the increased memory and latency issues of MoE models. We show that low-bit quantization together with the MoE architecture delivers a reliable model performance while reducing the memory size significantly even without any additional training in most cases. In particular, expert layers in MoE models are much more robust to the quantization than conventional feedforward networks (FFN) layers. In our comprehensive analysis, we show that MoE models with 2-bit expert weights can deliver better model performance than the dense model trained on the same dataset. As a result of low-bit quantization, we show the model size can be reduced by 79.6% of the original half precision floating point (fp16) MoE model. Combined with an optimized GPU runtime implementation, it also achieves 1.24X speed-up on A100 GPUs.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 12 of its 15 references that are in the map. First posted on arXiv 3 Oct 2023.

Builds on

12 of 15 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformers28 May
  2. 2021Efficient Large Scale Language Modeling with Mixtures of Experts20 Dec
  3. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference24 Sep
  4. 2021Scalable and Efficient MoE Training for Multitask Multilingual Models22 Sep
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun

Other references · 7

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  3. 2020On Layer Normalization in the Transformer Architecturecited by 906
  4. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  5. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  6. 2020Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translationcited by 104
  7. 2020Multi-task Learning for Multilingual Neural Machine Translationcited by 46

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 5; 2025: 14; 2026: 13.

34 papers.

Sort:

By area:

  1. 2023Fast Inference of Mixture-of-Experts Language Models with Offloadingcited by 88
  2. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Routercited by 58
  3. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  4. 2023QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Modelscited by 42
  5. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloadingcited by 24
  6. 2025Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipelinecited by 20
  7. 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Schedulingcited by 18
  8. 2025D2MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Servingcited by 17
  9. 2025Fate: Fast Edge Inference of Mixture-of-Experts Models via Cross-Layer Gatecited by 13
  10. 2025MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systemscited by 11
  11. 2025MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skippingcited by 11
  12. 2025EAQuant: Enhancing Post-Training Quantization for MoE Models via Expert-Aware Optimizationcited by 9
22 more
  1. 2026MoE-Spec: Expert Budgeting for Efficient Speculative Decodingcited by 9
  2. 2024Revisiting SMoE Language Models by Evaluating Inefficiencies with Task Specific Expert Pruningcited by 8
  3. 2024Mixture of Experts with Mixture of Precisions for Tuning Quality of Servicecited by 7
  4. 2025MiLo: Efficient Quantized MoE Inference with Mixture of Low-Rank Compensatorscited by 7
  5. 2025Dynamic Expert Quantization for Scalable Mixture-of-Experts Inferencecited by 6
  6. 2025CoServe: Efficient Collaboration-of-Experts (CoE) Model Inference with Limited Memorycited by 5
  7. 2026GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMscited by 5
  8. 2025Universality of Layer-Level Entropy-Weighted Quantization Beyond Model Architecture and Sizecited by 4
  9. 2025DynaMo: Runtime Switchable Quantization for MoE with Cross-Dataset Adaptationcited by 4
  10. 2026Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guaranteescited by 1
  11. 2026PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantizationcited by 1
  12. 2025SlimMoE: Structured Compression of Large MoE Models via Expert Slimming and Distillationcited by 0
  13. 2025Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devicescited by 0
  14. 2026ButterflyMoE: Compression-Scalable Ternary Experts via Structured Butterfly Orbitscited by 0
  15. 2026A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMscited by 0
  16. 2026TileQ: Efficient Low-Rank Quantization of Mixture-of-Experts with 2D Tilingcited by 0
  17. 2026Beyond Independent Optimization: Compression, MoE Routing, and Quantization Interactions in Multimodal Edge Intelligencecited by 0
  18. 2026From Expert Reduction to Behavioral Divergence: Tracing Numerical State through Sparse MoE Inferencecited by 0
  19. 2026Tied Trit-Planes: Constraining PTQTP to a Uniform Nine-Level Quantizer, with a Persistent Folded Format for Disk-Streamed Mixture-of-Experts Servingcited by 0
  20. 2026Every Expert Counts: ExactMoE for Memory-Efficient W4A16 Inferencecited by 0
  21. 2026Quality-Constrained Routing over a Fixed Pool of Quantized Mixture-of-Experts Instancescited by 0
  22. 2026Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Stagingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024QuantMoE-Bench: Examining Post-Training Quantization for Mixture-of-Experts
  2. 2025MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance
  3. 2025MoQE: Improve Quantization Model performance via Mixture of Quantization Experts
  4. 2025MoPEQ: Mixture of Mixed Precision Quantized Experts
  5. 2026KBVQ-MoE: KLT-guided SVD with Bias-Corrected Vector Quantization for MoE Large Language Models
  6. 2025Mixture of Experts in Large Language Models
  7. 2026AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantization
  8. 2025MC#: Mixture Compressor for Mixture-of-Experts Large Models