mlsanityData to 7 Oct 2026

Static Batching of Irregular Workloads on GPUs: Framework and Application to Efficient MoE Model Inference

arXiv 2501.16103 ↗First posted 27 Jan 2025cs.DC cs.LGcited by 2

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

It has long been a problem to arrange and execute irregular workloads on massively parallel devices. We propose a general framework for statically batching irregular workloads into a single kernel with a runtime task mapping mechanism on GPUs. We further apply this framework to Mixture-of-Experts (MoE) model inference and implement an optimized and efficient CUDA kernel. Our MoE kernel achieves up to 91% of the peak Tensor Core throughput on NVIDIA H800 GPU and 95% on NVIDIA H20 GPU.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 3 of its 19 references that are in the map. First posted on arXiv 27 Jan 2025.

Builds on

3 of 19 references.

References · 3

  1. 2024Mixtral of Expertscited by 1,785
  2. 2024DeepSeek-V3 Technical Reportcited by 724
  3. 2022A Review of Sparse Expert Models in Deep Learningcited by 147

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025SpecRouter: Adaptive Routing for Multi-Level Speculative Decoding in Large Language Modelscited by 2
  2. 2026Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Schedulingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026A Scheduling Framework for Efficient MoE Inference on Edge GPU-NDP Systems
  2. 2026GEM: GPU-Variability-Aware Expert to GPU Mapping for MoE Systems
  3. 2025HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference
  4. 2024DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inference
  5. 2026DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCs
  6. 2026Decoding the Skew: Distribution-Aware MoE Inference with Adaptive Kernel Dispatch
  7. 2025MoE-Gen: High-Throughput MoE Inference on a Single GPU with Module-Based Batching
  8. 2025HarMoEny: Efficient Multi-GPU Inference of MoE Models