# ParallelKittens: Systematic and Practical Simplification of Multi-GPU AI Kernels

arXiv: https://arxiv.org/abs/2511.13940 · HTML: https://mlsanity.com/paper/2511.13940/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Nov 2025.

## Built on by (most cited first, 12 of 12)

- [Composer 2 Technical Report](https://mlsanity.com/paper/2603.24477/) (2026)
- [MPK: A Compiler and Runtime for Mega-Kernelizing Tensor Programs](https://mlsanity.com/paper/2512.22219/) (2025)
- [CODA: Rewriting Transformer Blocks as GEMM-Epilogue Programs](https://mlsanity.com/paper/2605.19269/) (2026)
- [CUCo: An Agentic Framework for Compute and Communication Co-design](https://mlsanity.com/paper/2603.02376/) (2026)
- [Laguna M.1/XS.2 Technical Report](https://mlsanity.com/paper/2605.27605/) (2026)
- [CTA-Pipelining: A Latency-Oriented Spatial Scaling Method for Multi-GPU Systems](https://mlsanity.com/paper/2607.07862/) (2026)
- [X-Stage: Modeling Post-Issue Backpressure in GPU Communication--Computation Fusion](https://mlsanity.com/paper/2607.23264/) (2026)
- [mKernel: Fast Multi-GPU, Multi-Node Fused Kernels](https://mlsanity.com/paper/2609.13585/) (2026)
- [Mixture-of-Kittens: MoE Megakernel for NVL72s](https://mlsanity.com/paper/2609.36070/) (2026)
- [AI as a Compiler: Compiling Triton kernels without the Triton compiler](https://mlsanity.com/paper/2609.36800/) (2026)
- [Purlin: Separating Orchestration from the Datapath of Collectives](https://mlsanity.com/paper/2609.36954/) (2026)
- [T-CCL: Resource Efficient and Performant Collective Communication using Tensor Memory Accelerator](https://mlsanity.com/paper/2610.07098/) (2026)

## Abstract (arXiv)

Inter-GPU communication has become a major bottleneck for modern AI workloads as models scale and improvements in hardware compute throughput outpace improvements in interconnect bandwidth. Existing systems mitigate this through compute-communication overlap but often fail to meet theoretical peak performance across heterogeneous workloads and new accelerators. Instead of operator-specific techniques, we ask whether a small set of simple, reusable principles can systematically guide the design of optimal multi-GPU kernels. We present ParallelKittens (PK), a minimal CUDA framework that drastically simplifies the development of overlapped multi-GPU kernels. PK extends the ThunderKittens framework and embodies the principles of multi-GPU kernel design through eight core primitives and a unified programming template, derived from a comprehensive analysis of the factors that govern multi-GPU performanceunicodex2014data-transfer mechanisms, resource scheduling, and design overheads. We validate PK on both Hopper and Blackwell architectures. With fewer than 50 lines of device code, PK achieves up to 2.33 × speedup for data- and tensor-parallel workloads, 4.08 × for sequence-parallel workloads, and 1.22 × for expert-parallel workloads.