mlsanityData to 14 Oct 2026

MoQE: Improve Quantization Model performance via Mixture of Quantization Experts

arXiv 2508.09204 ↗First posted 9 Aug 2025cs.LG cs.AI cs.CVcited by 3

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Quantization method plays a crucial role in improving model efficiency and reducing deployment costs, enabling the widespread application of deep learning models on resource-constrained devices. However, the quantization process inevitably introduces accuracy degradation. In this paper, we propose Mixture of Quantization Experts( abbr. MoQE), a quantization inference framework based on the Mixture-of-Experts (MoE) architecture, aiming to jointly improve the performance of quantization models. MoQE combines multiple quantization variants of one full-precision model as specialized "quantization experts" and dynamically routes input data to the most suitable expert based on its characteristics. MoQE alleviates the performance degradation commonly seen in single quantization models through specialization quantization expert models. We design lightweight, structure-aware router models tailored for both CV and NLP tasks. Experimental evaluations on ResNet, LLaMA, and Qwen model families across benchmark datasets including ImageNet, WikiText, C4, and OpenWebText demonstrate that MoQE achieves performance comparable to SOTA quantization model, without incurring significant increases in inference latency.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 20 of its 27 references that are in the map. First posted on arXiv 9 Aug 2025.

Builds on

20 of 27 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints9 Dec
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 18

  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  2. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  3. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  4. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  5. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 561
  6. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
  7. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
  8. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 402
  9. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 278
  10. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 203
  11. 2020Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programmingcited by 92
  12. 2023Scaling Vision-Language Models with Sparse Mixture of Expertscited by 88
6 more
  1. 2021Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimationcited by 79
  2. 2022PD-Quant: Post-Training Quantization based on Prediction Difference Metriccited by 73
  3. 2023EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoEcited by 25
  4. 2023PaCE: Unified Multi-modal Dialogue Pre-training with Progressive and Compositional Expertscited by 19
  5. 2023CoT-MoTE: Exploring ConTextual Masked Auto-Encoder Pre-training with Mixture-of-Textual-Experts for Passage Retrievalcited by 7
  6. 2019DSConv: Efficient Convolution Operatorcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026VEQ: Modality-Adaptive Quantization for MoE Vision-Language Modelscited by 4
  2. 2026HeRo-Q: A General Framework for Stable Low Bit Quantization via Hessian Conditioningcited by 1
  3. 2026TileQ: Efficient Low-Rank Quantization of Mixture-of-Experts with 2D Tilingcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025MoPEQ: Mixture of Mixed Precision Quantized Experts
  2. 2023Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness
  3. 2025MoQAE: Mixed-Precision Quantization for Long-Context LLM Inference via Mixture of Quantization-Aware Experts
  4. 2024QuantMoE-Bench: Examining Post-Training Quantization for Mixture-of-Experts
  5. 2026GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs
  6. 2021MQBench: Towards Reproducible and Deployable Model Quantization Benchmark
  7. 2026Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees
  8. 2025MC#: Mixture Compressor for Mixture-of-Experts Large Models