mlsanityData to 7 Oct 2026

Mixture-of-Kittens: MoE Megakernel for NVL72s

arXiv 2609.36070 ↗First posted 28 Sep 2026cs.DC cs.LGcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

AI accelerator systems are rapidly consolidating into scale-up architectures, where tens to thousands of GPUs communicate over high-bandwidth, single-hop fabrics. We find that existing Mixture-of-Experts (MoE) training systems, optimized for conventional scale-out networks, transfer poorly to this setting, often running slower than a naive baseline built with PyTorch and NCCL. With industry roadmaps pointing toward even larger scale-up domains, understanding the performance tradeoffs of this hardware regime is increasingly important. We present Mixture-of-Kittens (MoK), an MoE training system designed for Nvidia NVL72. MoK builds on three insights that unlock performance on scale-up domains: (1) choosing push- or pull-based communication per operator, (2) restructuring the computation-communication overlap, and (3) fully eliminating CPU-GPU synchronization. MoK distills these insights into a single deterministic training megakernel that fuses token dispatch, shared and routed expert FFNs, and token combine. Across MoE layer shapes from four widely used open-weight models, MoK delivers up to 2.37× the throughput of the strongest publicly available baseline. In a production run on 512 GPUs spanning multiple GB300 NVL72 racks, MoK improves end-to-end training throughput by 1.41×.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 30 of its 50 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

30 of 50 references.

Closest ancestors · 9

Same direction, newest first

  1. 2026Scalable Training of Mixture-of-Experts Models with Megatron Core8 Mar
  2. 2025SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations16 Dec
  3. 2025FlashMoE: Fast Distributed MoE in a Single Kernel5 Jun
  4. 2025Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Experts27 Feb
  5. 2025FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models18 Jan
  6. 2024Scattered Mixture-of-Experts Implementation13 Mar
  7. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts29 Nov
  8. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  9. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 21

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  8. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  9. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 502
  10. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  11. 2018An Empirical Model of Large-Batch Trainingcited by 268
  12. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 198
9 more
  1. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
  2. 2023Microscaling Data Formats for Deep Learningcited by 132
  3. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughputcited by 85
  4. 2024FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusioncited by 53
  5. 2025Triton-distributed: Programming Overlapping Kernels on Distributed AI Systems with the Triton Compilercited by 24
  6. 2026Composer 2 Technical Reportcited by 20
  7. 2025MPK: A Compiler and Runtime for Mega-Kernelizing Tensor Programscited by 14
  8. 2025ParallelKittens: Systematic and Practical Simplification of Multi-GPU AI Kernelscited by 11
  9. 2026Demystifying NVSHMEM: A System-Level Analysis on Symmetric Memory and Device-Initiated Operations in GPU Communicationcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025X-MoE: Enabling Scalable Training for Emerging Mixture-of-Experts Architectures on HPC Platforms
  2. 2025MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production
  3. 2026Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Experts
  4. 2026Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models
  5. 2022HetuMoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training System
  6. 2026Expert Coupling in MoE Pretraining: Reducing All-to-All Overhead with Correlated Placement and Token Shuffling
  7. 2024Optimizing Mixture-of-Experts Inference Time Combining Model Deployment and Communication Scheduling
  8. 2025HeterMoE: Efficient Training of Mixture-of-Experts Models on Heterogeneous GPUs