mlsanityData to 7 Oct 2026

A Scheduling Framework for Efficient MoE Inference on Edge GPU-NDP Systems

arXiv 2601.03992 ↗First posted 7 Jan 2026cs.DC cs.AIcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture-of-Experts (MoE) models facilitate edge deployment by decoupling model capacity from active computation, yet their large memory footprint drives the need for GPU systems with near-data processing (NDP) capabilities that offload experts to dedicated processing units. However, deploying MoE models on such edge-based GPU-NDP systems faces three critical challenges: 1) severe load imbalance across NDP units due to non-uniform expert selection and expert parallelism, 2) insufficient GPU utilization during expert computation within NDP units, and 3) extensive data pre-profiling necessitated by unpredictable expert activation patterns for pre-fetching. To address these challenges, this paper proposes an efficient inference framework featuring three key optimizations. First, the underexplored tensor parallelism in MoE inference is exploited to partition and compute large expert parameters across multiple NDP units simultaneously towards edge low-batch scenarios. Second, a load-balancing-aware scheduling algorithm distributes expert computations across NDP units and GPU to maximize resource utilization. Third, a dataset-free pre-fetching strategy proactively loads frequently accessed experts to minimize activation delays. Experimental results show that our framework enables GPU-NDP systems to achieve 2.41x on average and up to 2.56x speedup in end-to-end latency compared to state-of-the-art approaches, significantly enhancing MoE inference efficiency in resource-constrained environments.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 30 of its 43 references that are in the map. First posted on arXiv 7 Jan 2026.

Builds on

30 of 43 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline9 Feb
  2. 2024DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inference16 Dec
  3. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs18 Nov
  4. 2024Duplex: A Device for Large Language Models with Mixture of Experts, Grouped Query Attention, and Continuous Batching2 Sep
  5. 2024MoNDE: Mixture of Near-Data Experts for Large-Scale Sparse Models29 May
  6. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models10 Feb
  7. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec

Other references · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  7. 2024Mixtral of Expertscited by 1,785
  8. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  9. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  10. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  11. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  12. 2019RecNMP: Accelerating Personalized Recommendation with Near-Memory Processingcited by 49
11 more
  1. 2019TensorDIMM: A Practical Near-Memory Processing Architecture for Embeddings and Tensor Operations in Deep Learningcited by 42
  2. 2024Phi-3 Safety Post-Training: Aligning Language Models with a "Break-Fix" Cyclecited by 32
  3. 2023Ramulator 2.0: A Modern, Modular, and Extensible DRAM Simulatorcited by 26
  4. 2023A Precision-Scalable RISC-V DNN Processor with On-Device Learning Capability at the Extreme Edgecited by 10
  5. 2025LP-Spec: Leveraging LPDDR PIM for Efficient LLM Mobile Speculative Inference with Architecture-Dataflow Co-Optimizationcited by 9
  6. 2024Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Formatcited by 8
  7. 2025Make LLM Inference Affordable to Everyone: Augmenting GPU Memory with NDP-DIMMcited by 6
  8. 2023BRAMAC: Compute-in-BRAM Architectures for Multiply-Accumulate on FPGAscited by 2
  9. 2024Co-Designing Binarized Transformer and Hardware Accelerator for Efficient End-to-End Edge Deploymentcited by 2
  10. 2023Near-Memory Parallel Indexing and Coalescing: Enabling Highly Efficient Indirect Access for SpMVcited by 1
  11. 2025NDPage: Efficient Address Translation for Near-Data Processing Architectures via Tailored Page Tablecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026TriMoE: Augmenting GPU with AMX-Enabled CPU and DIMM-NDP for High-Throughput MoE Inference via Offloading
  2. 2025Efficient MoE Inference with Fine-Grained Scheduling of Disaggregated Expert Parallelism
  3. 2025HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference
  4. 2026GEM: GPU-Variability-Aware Expert to GPU Mapping for MoE Systems
  5. 2025HarMoEny: Efficient Multi-GPU Inference of MoE Models
  6. 2026Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Staging
  7. 2025MoBiLE: Efficient Mixture-of-Experts Inference on Consumer GPU with Mixture of Big Little Experts
  8. 2025Accelerating Edge Inference for Distributed MoE Models with Latency-Optimized Expert Placement