mlsanityData to 14 Oct 2026

Uncertainty Makes It Stable: Curiosity-Driven Quantized Mixture-of-Experts

arXiv 2511.11743 ↗First posted 13 Nov 2025cs.LG cs.AIcited by 0

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Deploying deep neural networks on resource-constrained devices faces two critical challenges: maintaining accuracy under aggressive quantization while ensuring predictable inference latency. We present a curiosity-driven quantized Mixture-of-Experts framework that addresses both through Bayesian epistemic uncertainty-based routing across heterogeneous experts (BitNet ternary, 1-16 bit BitLinear, post-training quantization). Evaluated on audio classification benchmarks (ESC-50, Quinn, UrbanSound8K), our 4-bit quantization maintains 99.9 percent of full-precision F1 (0.858 vs 0.859) with 4x compression and 31 percent energy savings versus 8-bit, while both achieve statistical parity with full precision (p > 0.05). Crucially, curiosity-driven routing simultaneously improves accuracy and stability: on Quinn, F1 increases from 0.802 to 0.809 while cross-fold variance drops by 85 percent (p < 0.001, Levene's test), with reductions of 50 to 94 percent across datasets. The routing is self-organizing, with the high-precision 8-bit expert automatically receiving the most uncertain samples (20 percent lower confidence, p < 0.001), while lightweight experts handle easier inputs. Datasets with already low baseline variance show no artificial stability gain, confirming the mechanism targets genuine epistemic uncertainty rather than overfitting routing decisions. At 1.2M parameters, the framework provides interpretable, precision-aware routing suitable for safety-sensitive edge deployments where both accuracy and predictability are critical.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 15 of its 35 references that are in the map. First posted on arXiv 13 Nov 2025.

Builds on

15 of 35 references.

References · 15

  1. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  2. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,352
  3. 2019Searching for MobileNetV3cited by 1,994
  4. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  5. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  6. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  7. 2016CNN Architectures for Large-Scale Audio Classificationcited by 704
  8. 2019Learned Step Size Quantizationcited by 628
  9. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  10. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 426
  11. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
  12. 2023From Sparse to Soft Mixtures of Expertscited by 248
3 more
  1. 2019Memory- and Communication-Aware Model Compression for Distributed Deep Learning Inference on IoTcited by 13
  2. 20241-Bit FQT: Pushing the Limit of Fully Quantized Training to 1-bitcited by 3
  3. 2025Enhancing Uncertainty Estimation in Semantic Segmentation via Monte-Carlo Frequency Dropoutcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees
  2. 2025EAQuant: Enhancing Post-Training Quantization for MoE Models via Expert-Aware Optimization
  3. 2024On the Role of Discrete Representation in Sparse Mixture of Experts
  4. 2024QuantMoE-Bench: Examining Post-Training Quantization for Mixture-of-Experts
  5. 2026The qs Inequality: Quantifying the Double Penalty of Mixture-of-Experts at Inference
  6. 2026Value-and-Structure Alignment for Routing-Consistent Quantization of Mixture-of-Experts Models
  7. 2026Task-Phase-Conditioned Expert Routing Is Not a Reliable Proxy for Quantization Sensitivity in MoE FFN Tensors
  8. 2025MC#: Mixture Compressor for Mixture-of-Experts Large Models