# Static Batching of Irregular Workloads on GPUs: Framework and Application to Efficient MoE Model Inference

arXiv: https://arxiv.org/abs/2501.16103 · HTML: https://mlsanity.com/paper/2501.16103/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts.
- Builds on 3 of its 19 references that are in the map.
- First posted on arXiv 27 Jan 2025.

## Placed in

- Mixture-of-experts / [Efficient MoE training and inference systems](https://mlsanity.com/moe/efficient-moe-training-and-inference-systems/)

## Builds on (most cited first, 3 of 3)

- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [DeepSeek-V3 Technical Report](https://mlsanity.com/paper/2412.19437/) (2024)
- [A Review of Sparse Expert Models in Deep Learning](https://mlsanity.com/paper/2209.01667/) (2022)

## Built on by (most cited first, 2 of 2)

- [SpecRouter: Adaptive Routing for Multi-Level Speculative Decoding in Large Language Models](https://mlsanity.com/paper/2505.07680/) (2025)
- [Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Scheduling](https://mlsanity.com/paper/2603.27624/) (2026)

## Abstract (arXiv)

It has long been a problem to arrange and execute irregular workloads on massively parallel devices. We propose a general framework for statically batching irregular workloads into a single kernel with a runtime task mapping mechanism on GPUs. We further apply this framework to Mixture-of-Experts (MoE) model inference and implement an optimized and efficient CUDA kernel. Our MoE kernel achieves up to 91% of the peak Tensor Core throughput on NVIDIA H800 GPU and 95% on NVIDIA H20 GPU.