mlsanityData to 14 Oct 2026

Colla-Q: Toward Collaborative Experts in MoE Quantization via Minimax Precision Balancing

arXiv 2609.18131 ↗First posted 16 Sep 2026cs.LG cs.CLcited by 0

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

In this paper, we present a Mixture-of-Experts (MoE) quantization method based on activation entropy. Although quantization reduces memory and computational costs, it can substantially degrade performance. In particular, performance decline is pronounced in quantized MoE models, where individual experts have a small number of parameters that are sensitive to low-bit representation. Considering that MoE operates as an ensemble model with collaborative contributions from routed experts, a significant performance decline of a particular expert due to quantization can harm model performance. Therefore, we propose Colla-Q, a bit-allocation framework to maintain balanced performance across experts through an activation-entropy-based bit-width allocation algorithm. This approach encourages each expert to operate collaboratively in the quantized model, thereby 1) improving the overall MoE performance and 2) reducing the dependence on the calibration dataset. Since uniformly adjusting each expert's performance facilitates robustness and stability of the MoE model, the proposed MoE quantization method can generalize more consistently across different calibration datasets. Our code is available at: https://github.com/mmai-laboratory/Colla_Q

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 20 of its 28 references that are in the map. First posted on arXiv 16 Sep 2026.

Builds on

20 of 28 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design9 May
  2. 2024OLMoE: Open Mixture-of-Experts Language Models3 Sep
  3. 2024QuantMoE-Bench: Examining Post-Training Quantization for Mixture-of-Experts12 Jun
  4. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 15

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  9. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  10. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  11. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  12. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
3 more
  1. 2018Predictive Uncertainty Estimation via Prior Networkscited by 557
  2. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 402
  3. 2024Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensemblingcited by 28

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantization
  2. 2026Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees
  3. 2026Beyond Expert Rankings: Contrast Calibration for Mixed-Precision MoE Quantization
  4. 2025MC#: Mixture Compressor for Mixture-of-Experts Large Models
  5. 2025MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance
  6. 2026GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs
  7. 2026TileQ: Efficient Low-Rank Quantization of Mixture-of-Experts with 2D Tiling
  8. 2023Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness