# Optimal Software Pipelining and Warp Specialization for Tensor Core GPUs

arXiv: https://arxiv.org/abs/2512.18134 · HTML: https://mlsanity.com/paper/2512.18134/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 36 references that are in the map.
- First posted on arXiv 19 Dec 2025.

## Builds on (most cited first, 8 of 8)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [ThunderKittens: Simple, Fast, and Adorable AI Kernels](https://mlsanity.com/paper/2410.20399/) (2024)
- [Tawa: Automatic Warp Specialization for Modern GPUs with Asynchronous References](https://mlsanity.com/paper/2510.14719/) (2025)
- [ALCOP: Automatic Load-Compute Pipelining in Deep Learning Compiler for AI-GPUs](https://mlsanity.com/paper/2210.16691/) (2022)
- [Task-Based Tensor Computations on Modern GPUs](https://mlsanity.com/paper/2504.07004/) (2025)
- [The SCIP Optimization Suite 10.0](https://mlsanity.com/paper/2511.18580/) (2025)

## Built on by (most cited first, 5 of 5)

- [Magellan: Autonomous Discovery of Novel Compiler Optimization Heuristics with AlphaEvolve](https://mlsanity.com/paper/2601.21096/) (2026)
- [CAKE: Compiler-Agent Co-Design for Frontier Kernel Evolution](https://mlsanity.com/paper/2608.12629/) (2026)
- [GauS: Differentiable Scheduling Optimization via Gaussian Reparameterization](https://mlsanity.com/paper/2602.20427/) (2026)
- [Hardware-Aware Features for CUTLASS Kernel Selection](https://mlsanity.com/paper/2609.35587/) (2026)
- [RESOLVE: Language-Agnostic Validation of GPU Kernels Through Testing, Reduction, and Proof](https://mlsanity.com/paper/2610.05683/) (2026)

## Abstract (arXiv)

GPU architectures have continued to grow in complexity, with recent incarnations introducing increasingly powerful fixed-function units for matrix multiplication and data movement to accompany highly parallel general-purpose cores. To fully leverage these machines, software must use sophisticated schedules that maximally utilize all hardware resources. Since realizing such schedules is complex, both programmers and compilers routinely employ program transformations, such as software pipelining (SWP) and warp specialization (WS), to do so in practice. However, determining how best to use SWP and WS in combination is a challenging problem that is currently handled through a mix of brittle compilation heuristics and fallible human intuition, with little insight into the space of solutions. To remedy this situation, we introduce a novel formulation of SWP and WS as a joint optimization problem that can be solved holistically by off-the-shelf constraint solvers. We reify our approach in Twill, the first system that automatically derives optimal SWP and WS schedules for a large class of iterative programs. Twill is heuristic-free, easily extensible to new GPU architectures, and guaranteed to produce optimal schedules. We show that Twill can rediscover, and thereby prove optimal, the SWP and WS schedules manually developed by experts for Flash Attention on both the NVIDIA Hopper and Blackwell GPU architectures.