DynaNDE: Dynamic Near-Data Expert Scheduling for Batched MoE Inference
On the map: Mixture-of-experts → Efficient MoE training and inference systems
Abstract (arXiv)
Mixture-of-Experts (MoE) models enable efficient scaling of large language model (LLM) inference but suffer from substantial data-movement overhead when deployed on neural processing unit (NPU)-based systems. Near-Data Processing (NDP) provides a promising way to mitigate this bottleneck via cooperative NPU-NDP execution. However, existing NPU-NDP MoE systems do not fully account for hardware heterogeneity, dynamic expert-level concurrency, and temporal expert reuse during batched inference. This paper presents DynaNDE, a dynamic near-data expert scheduling framework that exploits NPU-NDP collaboration to accelerate batched MoE inference. DynaNDE introduces an analytical performance model that captures hardware heterogeneity, data-movement costs, and communication-computation overlap in cooperative NPU-NDP execution. Guided by this model, DynaNDE determines per-layer expert scheduling across the NPU and NDP while accounting for expert-level concurrency. DynaNDE also incorporates a reuse-aware runtime that avoids redundant parameter movement when experts reside in NPU memory. Experimental results show that DynaNDE achieves substantial throughput improvements over the state-of-the-art NPU-NDP MoE serving framework, with average speedups of 2.6× and 2.2× for the prefill and decoding stages, respectively.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 15 of its 34 references that are in the map. First posted on arXiv 31 Aug 2026.
Builds on
15 of 34 references.
Closest ancestors · 1
Same direction, newest first
- 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug
Other references · 14
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
- 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
- 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
- 2020Beyond English-Centric Multilingual Machine Translationcited by 559
- 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 381
- 2023LongCoder: A Long-Range Pre-trained Language Model for Code Completioncited by 110
- 2024NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencingcited by 48
- 2025PIM Is All You Need: A CXL-Enabled GPU-Free System for Large Language Model Inferencecited by 22
- 2019Processing Data Where It Makes Sense: Enabling In-Memory Computationcited by 21
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Fast MoE Inference via Predictive Prefetching and Expert Replication
- 2026Beyond Uniform Experts: Cost-Aware Expert Execution for Efficient Multi-Device MoE Inference
- 2024EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inference
- 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference
- 2025OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference
- 2025A3D-MoE: Acceleration of Large Language Models with Mixture of Experts via 3D Heterogeneous Integration
- 2026HDA-MoE: Hybrid Parallelism and Dynamic, Adaptive Scheduling for Mixture-of-Experts with 3D Near-Memory Processing
- 2025Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline