mlsanityData to 7 Oct 2026

DynaMo: Runtime Switchable Quantization for MoE with Cross-Dataset Adaptation

arXiv 2503.21135 ↗First posted 27 Mar 2025cs.LGcited by 4

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

As the Mix-of-Experts (MoE) architecture increases the number of parameters in large models, there is an even greater need for model quantization. However, existing quantization methods overlook the expert dynamics of MoE across multiple datasets. Moreover, the existing static quantization cannot adapt MoE to various data change scenarios. In this paper, we perform a multi-level analysis to reveal MoE dynamics and define the significance of each channel/each expert. Based on the analysis results, we propose textitDynaMo, an end-to-end MoE quantization framework. DynaMo adopts an expert-level mixed-precision baseline quantization strategy, which ensures the quantized MoEs are compatible with multiple existing datasets. Furthermore, DynaMo incorporates a channel-level dynamic switching mechanism to adapt these quantized MoE models to novel datasets. Experiments show that DynaMo achieves a 2.78~4.54 PPL decrease and a 1.85%~3.77% accuracy improvement in various datasets, with ~3x inference speedup and negligible overhead.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 23 of its 33 references that are in the map. First posted on arXiv 27 Mar 2025.

Builds on

23 of 33 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Models10 Sep
  2. 2024OLMoE: Open Mixture-of-Experts Language Models3 Sep
  3. 2024JetMoE: Reaching Llama2 Performance with 0.1M Dollars11 Apr
  4. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  5. 2023Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness3 Oct

Other references · 18

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  3. 2016Pointer Sentinel Mixture Modelscited by 2,992
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  5. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  6. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  7. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 426
  8. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
  9. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learningcited by 114
  10. 2023RPTQ: Reorder-based Post-training Quantization for Large Language Modelscited by 112
  11. 2018A Systematic Classification of Knowledge, Reasoning, and Context within the ARC Datasetcited by 54
  12. 2024AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inferencecited by 42
6 more
  1. 2024A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithmscited by 30
  2. 2023QuantEase: Optimization-based Quantization for Language Modelscited by 28
  3. 2024FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Expertscited by 27
  4. 2025FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Modelscited by 17
  5. 2025Mixture of Experts (MoE): A Big Data Perspectivecited by 17
  6. 2024HoME: Hierarchy of Multi-Gate Experts for Multi-Task Learning at Kuaishoucited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMscited by 5
  2. 2026AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantizationcited by 4
  3. 2026DynaGraph: Lightweight Multi-Model Interaction Framework via Dynamic Topological Reconfigurationcited by 0
  4. 2026MODE: Modality-Decomposed Expert-Level Mixed-Precision Quantization for MoE Multimodal LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024QuantMoE-Bench: Examining Post-Training Quantization for Mixture-of-Experts
  2. 2025MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design
  3. 2026Colla-Q: Toward Collaborative Experts in MoE Quantization via Minimax Precision Balancing
  4. 2025MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance
  5. 2026DyMoE: Dynamic Expert Orchestration with Mixed-Precision Quantization for Efficient MoE Inference on Edge
  6. 2022Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Production
  7. 2026BitsMoE: Efficient Spectral Energy-Guided Bit Allocation for MoE LLM Quantization
  8. 2026Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees