# Automatic Horizontal Fusion for GPU Kernels

arXiv: https://arxiv.org/abs/2007.01277 · HTML: https://mlsanity.com/paper/2007.01277/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jul 2020.

## Built on by (most cited first, 17 of 17)

- [NanoFlow: Towards Optimal Large Language Model Serving Throughput](https://mlsanity.com/paper/2408.12757/) (2024)
- [SparseTIR: Composable Abstractions for Sparse Compilation in Deep Learning](https://mlsanity.com/paper/2207.04606/) (2022)
- [POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference](https://mlsanity.com/paper/2410.18038/) (2024)
- [BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batching](https://mlsanity.com/paper/2412.03594/) (2024)
- [The CoRa Tensor Compiler: Compilation for Ragged Tensors with Minimal Padding](https://mlsanity.com/paper/2110.10221/) (2021)
- [Automated Pest Detection with DNN on the Edge for Precision Agriculture](https://mlsanity.com/paper/2108.00421/) (2021)
- [Event Tensor: A Unified Abstraction for Compiling Dynamic Megakernel](https://mlsanity.com/paper/2604.13327/) (2026)
- [Demystifying BERT: Implications for Accelerator Design](https://mlsanity.com/paper/2104.08335/) (2021)
- [LoRAFusion: Efficient LoRA Fine-Tuning for LLMs](https://mlsanity.com/paper/2510.00206/) (2025)
- [Collage: Seamless Integration of Deep Learning Backends with Automatic Placement](https://mlsanity.com/paper/2111.00655/) (2021)
- [Minuet: Accelerating 3D Sparse Convolutions on GPUs](https://mlsanity.com/paper/2401.06145/) (2023)
- [CUCo: An Agentic Framework for Compute and Communication Co-design](https://mlsanity.com/paper/2603.02376/) (2026)
- [Kernel Looping: Eliminating Synchronization Boundaries for Peak Inference Performance](https://mlsanity.com/paper/2410.23668/) (2024)
- [Efficient MoE Inference with Fine-Grained Scheduling of Disaggregated Expert Parallelism](https://mlsanity.com/paper/2512.21487/) (2025)
- [ALT: Boosting Deep Learning Performance by Breaking the Wall between Graph and Operator Level Optimizations](https://mlsanity.com/paper/2210.12415/) (2022)
- [CTA-Pipelining: A Latency-Oriented Spatial Scaling Method for Multi-GPU Systems](https://mlsanity.com/paper/2607.07862/) (2026)
- [KerColle: Unlocking Fine-Grained GPU Concurrency in Vision-Language-Action Models](https://mlsanity.com/paper/2609.22335/) (2026)

## Abstract (arXiv)

We present automatic horizontal fusion, a novel optimization technique that complements the standard kernel fusion techniques for GPU programs. Unlike the standard fusion, whose goal is to eliminate intermediate data round trips, our horizontal fusion technique aims to increase the thread-level parallelism to hide instruction latencies. We also present HFuse, a new source to source CUDA compiler that implements automatic horizontal fusion. Our experimental results show that horizontal fusion can speed up the running time by 2.5%-60.8%. Our results reveal that the horizontal fusion is especially beneficial for fusing kernels with instructions that require different kinds of GPU resources (e.g., a memory-intensive kernel and a compute-intensive kernel).