# DynaMo: Runtime Switchable Quantization for MoE with Cross-Dataset Adaptation

arXiv: https://arxiv.org/abs/2503.21135 · HTML: https://mlsanity.com/paper/2503.21135/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in MoE large language models, one of 12 directions in Mixture-of-experts.
- Builds on 23 of its 33 references that are in the map.
- First posted on arXiv 27 Mar 2025.

## Placed in

- Mixture-of-experts / [MoE large language models](https://mlsanity.com/moe/moe-large-language-models/)

## Builds on (most cited first, 23 of 23)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models](https://mlsanity.com/paper/2401.06066/) (2024)
- [HAQ: Hardware-Aware Automated Quantization with Mixed Precision](https://mlsanity.com/paper/1811.08886/) (2018)
- [OLMoE: Open Mixture-of-Experts Language Models](https://mlsanity.com/paper/2409.02060/) (2024)
- [Atom: Low-bit Quantization for Efficient and Accurate LLM Serving](https://mlsanity.com/paper/2310.19102/) (2023)
- [RPTQ: Reorder-based Post-training Quantization for Large Language Models](https://mlsanity.com/paper/2304.01089/) (2023)
- [DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning](https://mlsanity.com/paper/2106.03760/) (2021)
- [JetMoE: Reaching Llama2 Performance with 0.1M Dollars](https://mlsanity.com/paper/2404.07413/) (2024)
- [A Systematic Classification of Knowledge, Reasoning, and Context within the ARC Dataset](https://mlsanity.com/paper/1806.00358/) (2018)
- [AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference](https://mlsanity.com/paper/2408.10284/) (2024)
- [Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness](https://mlsanity.com/paper/2310.02410/) (2023)
- [A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithms](https://mlsanity.com/paper/2409.16694/) (2024)
- [QuantEase: Optimization-based Quantization for Language Models](https://mlsanity.com/paper/2309.01885/) (2023)
- [FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Experts](https://mlsanity.com/paper/2408.11304/) (2024)
- [Mixture of Experts (MoE): A Big Data Perspective](https://mlsanity.com/paper/2501.16352/) (2025)
- [FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models](https://mlsanity.com/paper/2501.10714/) (2025)
- [HoME: Hierarchy of Multi-Gate Experts for Multi-Task Learning at Kuaishou](https://mlsanity.com/paper/2408.05430/) (2024)
- [DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Models](https://mlsanity.com/paper/2409.06669/) (2024)

## Built on by (most cited first, 4 of 4)

- [GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs](https://mlsanity.com/paper/2605.23078/) (2026)
- [AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantization](https://mlsanity.com/paper/2606.04980/) (2026)
- [MODE: Modality-Decomposed Expert-Level Mixed-Precision Quantization for MoE Multimodal LLMs](https://mlsanity.com/paper/2606.17118/) (2026)
- [DynaGraph: Lightweight Multi-Model Interaction Framework via Dynamic Topological Reconfiguration](https://mlsanity.com/paper/2605.29511/) (2026)

## Abstract (arXiv)

As the Mix-of-Experts (MoE) architecture increases the number of parameters in large models, there is an even greater need for model quantization. However, existing quantization methods overlook the expert dynamics of MoE across multiple datasets. Moreover, the existing static quantization cannot adapt MoE to various data change scenarios. In this paper, we perform a multi-level analysis to reveal MoE dynamics and define the significance of each channel/each expert. Based on the analysis results, we propose textitDynaMo, an end-to-end MoE quantization framework. DynaMo adopts an expert-level mixed-precision baseline quantization strategy, which ensures the quantized MoEs are compatible with multiple existing datasets. Furthermore, DynaMo incorporates a channel-level dynamic switching mechanism to adapt these quantized MoE models to novel datasets. Experiments show that DynaMo achieves a 2.78~4.54 PPL decrease and a 1.85%~3.77% accuracy improvement in various datasets, with ~3x inference speedup and negligible overhead.