Decoding the Skew: Distribution-Aware MoE Inference with Adaptive Kernel Dispatch
On the map: Mixture-of-experts → Efficient MoE training and inference systems
Abstract (arXiv)
Mixture-of-Experts (MoE) inference consists of sparse expert GEMMs whose shapes vary with the runtime routing distribution. Existing serving systems typically select fused-MoE kernels using static token-count buckets, ignoring the per-expert routing distribution that determines tile padding, memory reuse, and kernel efficiency. We introduce a distribution-aware framework for modeling and benchmarking MoE inference. The framework combines the compact Effective Experts metric with a Dirichlet-based reverse-modeling procedure that generates controllable routing distributions for systematic hardware studies. Using it, we show that the best fused-MoE kernel changes with routing skew and token count. We further present DA-MoE, a GPU-resident kernel-dispatch runtime for NVIDIA GPUs that matches the live routing histogram to offline-tuned distributions and selects a near-optimal fused-MoE kernel without CPU--GPU synchronization. On HumanEval-X serving traces, DA-MoE improves geomean fused-MoE latency by 1.16X on DeepSeek-V3 and 1.29X on Kimi K2, with peak speedups of 1.40X and 1.56X.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 19 of its 27 references that are in the map. First posted on arXiv 25 Jul 2026.
Builds on
19 of 27 references.
Closest ancestors · 8
Same direction, newest first
- 2025Efficient CPU-GPU Collaborative Inference for MoE-based LLMs on Memory-Limited Systems18 Dec
- 2025SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations16 Dec
- 2025Patterns behind Chaos: Forecasting Data Movement for Efficient Large-Scale MoE LLM Inference7 Oct
- 2025X-MoE: Enabling Scalable Training for Emerging Mixture-of-Experts Architectures on HPC Platforms18 Aug
- 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs18 Nov
- 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices28 Aug
- 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts29 Nov
- 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct
Other references · 11
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2025Qwen3 Technical Reportcited by 8,195
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
- 2024Mixtral of Expertscited by 1,785
- 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
- 2025Kimi K2: Open Agentic Intelligencecited by 523
- 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 294
- 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
- 2024Turn Waste into Worth: Rectifying Top-k Router of MoEcited by 11
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026GEM: GPU-Variability-Aware Expert to GPU Mapping for MoE Systems
- 2026A Scheduling Framework for Efficient MoE Inference on Edge GPU-NDP Systems
- 2025Accelerating MoE Model Inference with Expert Sharding
- 2026Cross-Platform Fused MoE Dispatch in Triton: Portable Expert Routing Without CUDA
- 2026Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Staging
- 2025Static Batching of Irregular Workloads on GPUs: Framework and Application to Efficient MoE Model Inference
- 2025ExpertFlow: Adaptive Expert Scheduling and Memory Coordination for Efficient MoE Inference
- 2024DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inference