mlsanityData to 14 Oct 2026

Beyond Expert Rankings: Contrast Calibration for Mixed-Precision MoE Quantization

arXiv 2610.20353 ↗First posted 14 Oct 2026cs.LG stat.MLcited by 0

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Mixture-of-experts (MoE) language models activate only a few experts per token, yet all expert weights must be stored. Mixed-precision quantization reduces this storage cost by assigning higher bit-widths to sensitive experts and lower bit-widths to resilient ones, but expert rankings alone do not determine how strongly to weight their errors. To address this gap, we propose NatScale, a contrast-calibrated allocation framework for mixed-precision MoE quantization. It uses a model-specific power transformation to calibrate gradient-based expert weights to an information-theoretic contrast target while preserving rankings. We jointly evaluate each expert's gate, up and down configurations to estimate its output error. We then combine these errors and calibrated weights with an existing prior based on weight spectra, variance and precision decay to select configurations under a storage budget. Across three models, NatScale achieves the lowest native-to-quantized KL divergence and the highest top-token agreement in all nine settings against five allocation baselines. It also yields the lowest perplexity in seven settings. These findings highlight the importance of objective-aware precision allocation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 29 of its 42 references that are in the map. First posted on arXiv 14 Oct 2026.

Builds on

29 of 42 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026Q-Strata: Hierarchical Bit Allocation for Mixed-Precision Quantization of Mixture-of-Experts LLMs31 Aug
  2. 2026AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantization3 Jun
  3. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence26 Apr
  4. 2025CAMERA: Multi-Matrix Joint Compression for MoE Models via Micro-Expert Redundancy Analysis4 Aug
  5. 2025MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design9 May
  6. 2025MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance2 May
  7. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model7 May

Other references · 22

  1. 2016Pointer Sentinel Mixture Modelscited by 3,096
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  4. 2021Convex optimizationcited by 2,515
  5. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  7. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  8. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
  9. 2018Hierarchical Neural Story Generationcited by 1,193
  10. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  11. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
  12. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
10 more
  1. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  2. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 297
  3. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
  4. 2023On the Impact of Calibration Data in Post-training Quantization and Pruningcited by 58
  5. 2025GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidancecited by 32
  6. 2024Self-calibration for Language Model Quantization and Pruningcited by 13
  7. 2026GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Samplingcited by 6
  8. 2026KronQ: LLM Quantization via Kronecker-Factored Hessiancited by 3
  9. 2022Maximum likelihood estimation of the Weibull distribution with reduced biascited by 1
  10. 2026Robust Ultra Low-Bit Post-Training Quantization via Stable Diagonal Curvature Estimatecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024QuantMoE-Bench: Examining Post-Training Quantization for Mixture-of-Experts
  2. 2026Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees
  3. 2026GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs
  4. 2023Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness
  5. 2026BitsMoE: Efficient Spectral Energy-Guided Bit Allocation for MoE LLM Quantization
  6. 2026Colla-Q: Toward Collaborative Experts in MoE Quantization via Minimax Precision Balancing
  7. 2025MoPEQ: Mixture of Mixed Precision Quantized Experts
  8. 2024Mixture Compressor for Mixture-of-Experts LLMs Gains More