mlsanityData to 14 Oct 2026

Efficient Mixture-of-Experts LLM Inference with Apple Silicon NPUs

arXiv 2604.18788 ↗First posted 20 Apr 2026cs.LGcited by 2

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Apple Neural Engine (ANE) is a dedicated neural processing unit (NPU) present in every Apple Silicon chip. Mixture-of-Experts (MoE) LLMs improve inference efficiency via sparse activation but are challenging for NPUs in three ways: expert routing is unpredictable and introduces dynamic tensor shapes that conflict with the shape-specific constraints of NPUs; several irregular operators, e.g., top-k, scatter/gather, etc., are not NPU-friendly; and launching many small expert kernels incurs substantial dispatch and synchronization overhead. NPUs are designed to offload AI compute from CPU and GPU; our goal is to enable such offloading for MoE inference, particularly during prefill, where long-context workloads consume substantial system resources. This paper presents NPUMoE, a runtime inference engine that accelerates MoE execution on Apple Silicon by offloading dense, static computation to NPU, while preserving a CPU/GPU fallback path for dynamic operations. NPUMoE uses offline calibration to estimate expert capacity and popularity that drives three key techniques: (1) Static tiers for expert capacity to address dynamic expert routing (2) Grouped expert execution to mitigate NPU concurrency limits (3) Load-aware expert compute graph residency to reduce CPU-NPU synchronization overhead. Experiments on Apple M-series devices using three representative MoE LLMs and four long-context workloads show that NPUMoE consistently outperforms baselines, reducing latency by 1.32x-5.55x, improving energy efficiency by 1.81x-7.37x, and reducing CPU-cycle usage by 1.78x-5.54x through effective NPU offloading.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 19 of its 55 references that are in the map. First posted on arXiv 20 Apr 2026.

Builds on

19 of 55 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025LayerScope: Predictive Cross-Layer Scheduling for Efficient Multi-Batch MoE Inference on Legacy Servers28 Sep
  2. 2025Serving Large Language Models on Huawei CloudMatrix38415 Jun
  3. 2025FlashMoE: Fast Distributed MoE in a Single Kernel5 Jun
  4. 2024MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache25 Jan

Other references · 15

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  5. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  6. 2024DeepSeek-V3 Technical Reportcited by 811
  7. 2024Phi-4 Technical Reportcited by 711
  8. 2025Kimi-VL Technical Reportcited by 388
  9. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 193
  10. 2025Agent.xpu: Efficient Scheduling of Agentic LLM Workloads on Heterogeneous SoCcited by 11
  11. 2025Scaling LLM Test-Time Compute with Mobile NPU on Smartphonescited by 11
  12. 2025Accelerating Mobile Language Model via Speculative Decoding and NPU-Coordinated Executioncited by 5
3 more
  1. 2025ShadowNPU: System and Algorithm Co-design for NPU-Centric On-Device LLM Inferencecited by 4
  2. 2025Huawei Cloud Model-as-a-Service on the CloudMatrix384 SuperPodcited by 3
  3. 2026Accelerating OpenPangu Inference on NPU via Speculative Decodingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026How Weight Encoding Affects Language Model Placement and Performance on the Apple Neural Enginecited by 0
  2. 2026EStream: Fast and Memory-Efficient MoE Prefill through Expert Virtualization on Mobile NPUscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026TriMoE: Augmenting GPU with AMX-Enabled CPU and DIMM-NDP for High-Throughput MoE Inference via Offloading
  2. 2025FloE: On-the-Fly MoE Inference on Memory-constrained GPU
  3. 2025A3D-MoE: Acceleration of Large Language Models with Mixture of Experts via 3D Heterogeneous Integration
  4. 2026DynaNDE: Dynamic Near-Data Expert Scheduling for Batched MoE Inference
  5. 2026Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study
  6. 2026Beyond Uniform Experts: Cost-Aware Expert Execution for Efficient Multi-Device MoE Inference
  7. 2026FORGE: Procedural Weight Generation for Sparse MoE Inference
  8. 2025MoE-Lens: Towards the Hardware Limit of High-Throughput MoE LLM Serving Under Resource Constraints