mlsanityData to 14 Oct 2026

MoPEQ: Mixture of Mixed Precision Quantized Experts

arXiv 2509.02512 ↗First posted 2 Sep 2025cs.LGcited by 5

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Large Language and Vision Models using a Mixture-of-Experts (MoE) architecture pose significant challenges for deployment due to their computational and memory demands. Mixed Precision Quantization assigns different precisions to different layers of an LLM/VLM based on layer sensitivity and importance within the model. In this work, we propose a Post Training Quantization algorithm, MoPEQ, that assigns optimal bit width to each expert. Our method balances accuracy and model size by analyzing each expert's sensitivity using Hessian trace approximation instead of relying on the activation frequency of the expert. This per-expert granularity approach clusters similar experts to maintain model performance while reducing memory requirements. The experimental results on VLMEvalKit benchmark datasets using State-of-the-art VLMs Deepseek-VL2 -tiny, -small, -base, and MolmoE models demonstrate that our mixed precision quantized MoEs achieve competitive accuracy with substantial improvements in memory footprint compared to uniform-precision baseline methods. We perform a comprehensive study to analyze the impact of expert activation frequency and sensitivity using Hessian trace approximation at both layer-wise and model-wide expert precision allocation of 2, 3, and 4 bits to provide a thorough understanding of mixed precision quantization of VLM-MoEs.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 34 of its 43 references that are in the map. First posted on arXiv 2 Sep 2025.

Builds on

34 of 43 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Mixture Compressor for Mixture-of-Experts LLMs Gains More8 Oct
  2. 2024OLMoE: Open Mixture-of-Experts Language Models3 Sep
  3. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model7 May
  4. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  5. 2024Mixtral of Experts8 Jan
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  6. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  8. 2019Towards VQA Models That Can Readcited by 1,718
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  10. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  11. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  12. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
16 more
  1. 2023A Survey on Multimodal Large Language Modelscited by 909
  2. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  3. 2016A Diagram Is Worth A Dozen Imagescited by 804
  4. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 572
  5. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  6. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  7. 2021InfographicVQAcited by 437
  8. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 402
  9. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
  10. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 300
  11. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 292
  12. 2023A Survey on Model Compression for Large Language Modelscited by 269
  13. 2019HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networkscited by 261
  14. 2023Efficient Large Language Models: A Surveycited by 186
  15. 2023Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMscited by 50
  16. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 47

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2026GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Samplingcited by 6
  2. 2025Bandwidth-Efficient Adaptive Mixture-of-Experts via Low-Rank Compensationcited by 3
  3. 2026Noesis: Bidirectional Graph-RAG with Adaptive Parallelism and Cross-Knowledge-Base Semantic Discoverycited by 1
  4. 2025SignRoundV2: Toward Closing the Performance Gap in Extremely Low-Bit Post-Training Quantization for LLMscited by 0
  5. 2026DuoRate: Two-Rate Per-Expert Precision Allocation for Mixture-of-Experts Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees
  2. 2025MoQE: Improve Quantization Model performance via Mixture of Quantization Experts
  3. 2023Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness
  4. 2025MC#: Mixture Compressor for Mixture-of-Experts Large Models
  5. 2024QuantMoE-Bench: Examining Post-Training Quantization for Mixture-of-Experts
  6. 2026AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantization
  7. 2026GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs
  8. 2026VEQ: Modality-Adaptive Quantization for MoE Vision-Language Models