mlsanityData to 7 Oct 2026

DyMoE: Dynamic Expert Orchestration with Mixed-Precision Quantization for Efficient MoE Inference on Edge

arXiv 2603.19172 ↗First posted 19 Mar 2026cs.LGcited by 2

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Despite the computational efficiency of MoE models, the excessive memory footprint and I/O overhead inherent in multi-expert architectures pose formidable challenges for real-time inference on resource-constrained edge platforms. While existing static methods struggle with a rigid latency-accuracy trade-off, we observe that expert importance is highly skewed and depth-dependent. Motivated by these insights, we propose DyMoE, a dynamic mixed-precision quantization framework designed for high-performance edge inference. Leveraging insights into expert importance skewness and depth-dependent sensitivity, DyMoE introduces: (1) importance-aware prioritization to dynamically quantize experts at runtime; (2) depth-adaptive scheduling to preserve semantic integrity in critical layers; and (3) look-ahead prefetching to overlap I/O stalls. Experimental results on commercial edge hardware show that DyMoE reduces Time-to-First-Token (TTFT) by 3.44x-22.7x and up to a 14.58x speedup in Time-Per-Output-Token (TPOT) compared to state-of-the-art offloading baselines, enabling real-time, accuracy-preserving MoE inference on resource-constrained edge devices.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 26 of its 33 references that are in the map. First posted on arXiv 19 Mar 2026.

Builds on

26 of 33 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025FloE: On-the-Fly MoE Inference on Memory-constrained GPU9 May
  2. 2025Fate: Fast Edge Inference of Mixture-of-Experts Models via Cross-Layer Gate17 Feb
  3. 2025Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline9 Feb
  4. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs18 Nov
  5. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference3 Nov
  6. 2024AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference19 Aug
  7. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models10 Feb
  8. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec
  9. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices28 Aug

Other references · 17

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  6. 2024Mixtral of Expertscited by 1,785
  7. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  8. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  9. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  10. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  11. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inferencecited by 148
  12. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 140
5 more
  1. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  2. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 104
  3. 2024On-Device Language Models: A Comprehensive Reviewcited by 101
  4. 2024A Review on Edge Large Language Models: Design, Execution, and Applicationscited by 98
  5. 2025SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Servingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026WiSP: A Working-Set View of Mixture-of-Experts Serving on Extremely Low-Resource Hardwarecited by 1
  2. 2026PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantizationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026APEX: Adaptive Expert Prefetching for Memory-Efficient Edge MoE Inference
  2. 2025Dynamic Expert Quantization for Scalable Mixture-of-Experts Inference
  3. 2026Beyond Uniform Experts: Cost-Aware Expert Execution for Efficient Multi-Device MoE Inference
  4. 2026Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Scheduling
  5. 2025Efficient Edge LLMs Deployment via HessianAware Quantization and CPU GPU Collaborative
  6. 2025SliceMoE: Bit-Sliced Expert Caching under Miss-Rate Constraints for Efficient MoE Inference
  7. 2025OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference
  8. 2026DynaNDE: Dynamic Near-Data Expert Scheduling for Batched MoE Inference