mlsanityData to 14 Oct 2026

Tile-Level Activation Overlap for Efficient LLM Inference

arXiv 2607.02521 ↗First posted 5 May 2026cs.DC cs.LG cs.PLcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

SwiGLU is the dominant MLP activation in modern large language models, yet its intermediate tensor materialization costs 9-37% of MLP execution time. We present two complementary CUTLASS-based SM90 kernels that fuse SwiGLU into GeMM at the tile level. Kernel-1 overlaps Swish computation on the Gate accumulator with Up-tile loading using the Pingpong warp-specialized schedule; Kernel-2 interleaves SwiGLU with tile stores via a custom Epilogue Visitor Tree. Evaluated on Qwen-2.5 models (0.5B-72B) on NVIDIA H100, our kernels achieve up to 2.47x speedup over PyTorch, shifting workloads from memory-bound to compute-bound and reaching 79.5% peak BF16 utilization. We demonstrate that torch.compile cannot replicate this fusion (3-7x slower than our kernels), validating the need for hand-crafted tile-level design. Our fused kernels are also numerically superior, achieving zero mismatches compared to 4.5-11% for cuBLAS.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 7 of its 10 references that are in the map. First posted on arXiv 5 May 2026.

Builds on

7 of 10 references.

References · 7

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  5. 2024Qwen2 Technical Reportcited by 2,120
  6. 2020GLU Variants Improve Transformercited by 1,804
  7. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026BaseRT: Advancing Best-in-Class LLM Inference with Apple M5 Neural Accelerators
  2. 2025Efficient LLM Inference with Activation Checkpointing and Hybrid Caching
  3. 2026Ragged Paged Attention: A High-Performance and Flexible LLM Inference Kernel for TPU
  4. 2026SMEPilot: Characterizing and Optimizing LLM Inference with Scalable Matrix Extensions
  5. 2026Scaling LLM Inference Beyond Amdahl`s Limits via Eliminating Non-Scalable Overheads ↗
  6. 2026Deep Kernel Fusion for Transformers
  7. 2024LUT Tensor Core: A Software-Hardware Co-Design for LUT-Based Low-Bit LLM Inference
  8. 2024QUICK: Quantization-aware Interleaving and Conflict-free Kernel for efficient LLM inference