# D2MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Serving

arXiv: https://arxiv.org/abs/2504.15299 · HTML: https://mlsanity.com/paper/2504.15299/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts.
- Builds on 24 of its 40 references that are in the map.
- First posted on arXiv 17 Apr 2025.

## Placed in

- Mixture-of-experts / [Efficient MoE training and inference systems](https://mlsanity.com/moe/efficient-moe-training-and-inference-systems/)
- Efficient LLM inference and serving / [Diffusion LLM and MoE inference](https://mlsanity.com/inference/diffusion-llm-and-moe-inference/)

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU](https://mlsanity.com/paper/2312.12456/) (2023)
- [Llumnix: Dynamic Scheduling for Large Language Model Serving](https://mlsanity.com/paper/2406.03243/) (2024)
- [LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training](https://mlsanity.com/paper/2406.16554/) (2024)
- [MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs](https://mlsanity.com/paper/2411.11217/) (2024)
- [Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models](https://mlsanity.com/paper/2402.07033/) (2024)
- [AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference](https://mlsanity.com/paper/2408.10284/) (2024)
- [Mixture Compressor for Mixture-of-Experts LLMs Gains More](https://mlsanity.com/paper/2410.06270/) (2024)
- [Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness](https://mlsanity.com/paper/2310.02410/) (2023)
- [Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules](https://mlsanity.com/paper/2407.06677/) (2024)
- [OTAS: An Elastic Transformer Serving System via Token Adaptation](https://mlsanity.com/paper/2401.05031/) (2024)
- [Minimum Viable Device Drivers for ARM TrustZone](https://mlsanity.com/paper/2110.08303/) (2021)

## Built on by (most cited first, 20 of 20)

- [ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Scheduling](https://mlsanity.com/paper/2601.21198/) (2026)
- [EC2MoE: Adaptive End-Cloud Pipeline Collaboration Enabling Scalable Mixture-of-Experts Inference](https://mlsanity.com/paper/2508.06024/) (2025)
- [SMoE: An Algorithm-System Co-Design for Pushing MoE to the Edge via Expert Substitution](https://mlsanity.com/paper/2508.18983/) (2025)
- [SliceMoE: Bit-Sliced Expert Caching under Miss-Rate Constraints for Efficient MoE Inference](https://mlsanity.com/paper/2512.12990/) (2025)
- [MatGPTQ: Efficient and Accurate Inference over Nested Quantized Models](https://mlsanity.com/paper/2602.03537/) (2026)
- [PPAI: Enabling Personalized LLM Agent Interoperability for Collaborative Edge Intelligence](https://mlsanity.com/paper/2605.18067/) (2026)
- [TwinQuant: Learnable Subspace Decomposition for 4-Bit LLM Quantization](https://mlsanity.com/paper/2606.01556/) (2026)
- [MosaicQuant: Inlier-Outlier Disaggregation for Unified 4-Bit LLM Quantization](https://mlsanity.com/paper/2606.15652/) (2026)
- [MoE-Prism: Disentangling Monolithic Experts for Elastic MoE Services via Model-System Co-Designs](https://mlsanity.com/paper/2510.19366/) (2025)
- [MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Any-Precision LLM](https://mlsanity.com/paper/2602.20191/) (2026)
- [StagQ: Constraint-Driven Multi-Precision Weight Quantization for LLMs](https://mlsanity.com/paper/2610.05977/) (2026)
- [FlexMoRE: A Flexible Mixture of Rank-heterogeneous Experts for Efficient Federatedly-trained Large Language Models](https://mlsanity.com/paper/2602.08818/) (2026)
- [Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Scheduling](https://mlsanity.com/paper/2603.27624/) (2026)
- [Director: Accelerating Distributed MoE Serving via Online Proactive Expert Placement](https://mlsanity.com/paper/2607.08782/) (2026)
- [Beyond Independent Optimization: Compression, MoE Routing, and Quantization Interactions in Multimodal Edge Intelligence](https://mlsanity.com/paper/2607.20981/) (2026)
- [HetRoute Heterogeneous and Cost-aware Collaborative Routing Framework for Distributed Edge MoE Inference](https://mlsanity.com/paper/2608.00577/) (2026)
- [LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inference](https://mlsanity.com/paper/2609.03079/) (2026)
- [EStream: Fast and Memory-Efficient MoE Prefill through Expert Virtualization on Mobile NPUs](https://mlsanity.com/paper/2609.06551/) (2026)
- [Task-Aware Federated Fine-Tuning for MoE-based Large Language Models](https://mlsanity.com/paper/2609.13395/) (2026)
- [DuoRate: Two-Rate Per-Expert Precision Allocation for Mixture-of-Experts Serving](https://mlsanity.com/paper/2610.13328/) (2026)

## Abstract (arXiv)

The mixture of experts (MoE) model is a sparse variant of large language models (LLMs), designed to hold a better balance between intelligent capability and computational overhead. Despite its benefits, MoE is still too expensive to deploy on resource-constrained edge devices, especially with the demands of on-device inference services. Recent research efforts often apply model compression techniques, such as quantization, pruning and merging, to restrict MoE complexity. Unfortunately, due to their predefined static model optimization strategies, they cannot always achieve the desired quality-overhead trade-off when handling multiple requests, finally degrading the on-device quality of service. These limitations motivate us to propose the D2MoE, an algorithm-system co-design framework that matches diverse task requirements by dynamically allocating the most proper bit-width to each expert. Specifically, inspired by the nested structure of matryoshka dolls, we propose the matryoshka weight quantization (MWQ) to progressively compress expert weights in a bit-nested manner and reduce the required runtime memory. On top of it, we further optimize the I/O-computation pipeline and design a heuristic scheduling algorithm following our hottest-expert-bit-first (HEBF) principle, which maximizes the expert parallelism between I/O and computation queue under constrained memory budgets, thus significantly reducing the idle temporal bubbles waiting for the experts to load. Evaluations on real edge devices show that D2MoE improves the overall inference throughput by up to 1.39× and reduces the peak memory footprint by up to 53% over the latest on-device inference frameworks, while still preserving comparable serving accuracy as its INT8 counterparts.