# DNNFusion: Accelerating Deep Neural Networks Execution with Advanced Operator Fusion

arXiv: https://arxiv.org/abs/2108.13342 · HTML: https://mlsanity.com/paper/2108.13342/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 114 references that are in the map.
- First posted on arXiv 30 Aug 2021.

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [Learning to Optimize Tensor Programs](https://mlsanity.com/paper/1805.08166/) (2018)
- [Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions](https://mlsanity.com/paper/1802.04730/) (2018)
- [PatDNN: Achieving Real-Time DNN Execution on Mobile Devices with Pattern-based Weight Pruning](https://mlsanity.com/paper/2001.00138/) (2020)
- [MLIR: A Compiler Infrastructure for the End of Moore's Law](https://mlsanity.com/paper/2002.11054/) (2020)
- [MNN: A Universal and Efficient Inference Engine](https://mlsanity.com/paper/2002.12418/) (2020)
- [Tiramisu: A Polyhedral Compiler for Expressing Fast and Portable Code](https://mlsanity.com/paper/1804.10694/) (2018)
- [RTMobile: Beyond Real-Time Mobile Acceleration of RNNs for Speech Recognition](https://mlsanity.com/paper/2002.11474/) (2020)
- [Cortex: A Compiler for Recursive Deep Learning Models](https://mlsanity.com/paper/2011.01383/) (2020)
- [On Optimizing Operator Fusion Plans for Large-Scale Machine Learning in SystemML](https://mlsanity.com/paper/1801.00829/) (2018)
- [Polly's Polyhedral Scheduling in the Presence of Reductions](https://mlsanity.com/paper/1505.07716/) (2015)

## Built on by (most cited first, 30 of 57)

- [Model Compression and Efficient Inference for Large Language Models: A Survey](https://mlsanity.com/paper/2402.09748/) (2024)
- [Pre-RMSNorm and Pre-CRMSNorm Transformers: Equivalent and Efficient Pre-LN Transformers](https://mlsanity.com/paper/2305.14858/) (2023)
- [Understanding GNN Computational Graph: A Coordinated Computation, IO, and Memory Perspective](https://mlsanity.com/paper/2110.09524/) (2021)
- [M-ANT: Efficient Low-bit Group Quantization for LLMs via Mathematically Adaptive Numerical Type](https://mlsanity.com/paper/2502.18755/) (2025)
- [DreamShard: Generalizable Embedding Table Placement for Recommender Systems](https://mlsanity.com/paper/2210.02023/) (2022)
- [Towards Artificial General Intelligence (AGI) in the Internet of Things (IoT): Opportunities and Challenges](https://mlsanity.com/paper/2309.07438/) (2023)
- [SmartMem: Layout Transformation Elimination and Adaptation for Efficient DNN Execution on Mobile](https://mlsanity.com/paper/2404.13528/) (2024)
- [Operator Fusion in XLA: Analysis and Evaluation](https://mlsanity.com/paper/2301.13062/) (2023)
- [Enabling Resource-efficient AIoT System with Cross-level Optimization: A survey](https://mlsanity.com/paper/2309.15467/) (2023)
- [Neuromorphic Principles for Efficient Large Language Models on Intel Loihi 2](https://mlsanity.com/paper/2503.18002/) (2025)
- [Optimal Kernel Orchestration for Tensor Programs with Korch](https://mlsanity.com/paper/2406.09465/) (2024)
- [OLLA: Optimizing the Lifetime and Location of Arrays to Reduce the Memory Usage of Neural Networks](https://mlsanity.com/paper/2210.12924/) (2022)
- [Shared Memory-contention-aware Concurrent DNN Execution for Diversely Heterogeneous System-on-Chips](https://mlsanity.com/paper/2308.05869/) (2023)
- [SoD2: Statically Optimizing Dynamic Deep Neural Network](https://mlsanity.com/paper/2403.00176/) (2024)
- [Understanding the Performance Horizon of the Latest ML Workloads with NonGEMM Workloads](https://mlsanity.com/paper/2404.11788/) (2024)
- [Neptune: Advanced ML Operator Fusion for Locality and Parallelism on GPUs](https://mlsanity.com/paper/2510.08726/) (2025)
- [Event Tensor: A Unified Abstraction for Compiling Dynamic Megakernel](https://mlsanity.com/paper/2604.13327/) (2026)
- [Efficient Adaptive Activation Rounding for Post-Training Quantization](https://mlsanity.com/paper/2208.11945/) (2022)
- [MTP: A Meaning-Typed Language Abstraction for AI-Integrated Programming](https://mlsanity.com/paper/2405.08965/) (2024)
- [Survey for Landing Generative AI in Social and E-commerce Recsys -- the Industry Perspectives](https://mlsanity.com/paper/2406.06475/) (2024)
- [msf-CNN: Patch-based Multi-Stage Fusion with Convolutional Neural Networks for TinyML](https://mlsanity.com/paper/2505.11483/) (2025)
- [LoRAFusion: Efficient LoRA Fine-Tuning for LLMs](https://mlsanity.com/paper/2510.00206/) (2025)
- [Collage: Seamless Integration of Deep Learning Backends with Automatic Placement](https://mlsanity.com/paper/2111.00655/) (2021)
- [DNNFuser: Generative Pre-Trained Transformer as a Generalized Mapper for Layer Fusion in DNN Accelerators](https://mlsanity.com/paper/2201.11218/) (2022)
- [TorchBench: Benchmarking PyTorch with High API Surface Coverage](https://mlsanity.com/paper/2304.14226/) (2023)
- [Optimizing 3D Gaussian Splattering for Mobile GPUs](https://mlsanity.com/paper/2511.16298/) (2025)
- [OLLIE: Derivation-based Tensor Program Optimizer](https://mlsanity.com/paper/2208.02025/) (2022)
- [Inference Latency Prediction at the Edge](https://mlsanity.com/paper/2210.02620/) (2022)
- [AGO: Boosting Mobile AI Inference Performance by Removing Constraints on Graph Optimization](https://mlsanity.com/paper/2212.01005/) (2022)
- [RAF: Holistic Compilation for Deep Learning Model Training](https://mlsanity.com/paper/2303.04759/) (2023)

## Abstract (arXiv)

Deep Neural Networks (DNNs) have emerged as the core enabler of many major applications on mobile devices. To achieve high accuracy, DNN models have become increasingly deep with hundreds or even thousands of operator layers, leading to high memory and computational requirements for inference. Operator fusion (or kernel/layer fusion) is key optimization in many state-of-the-art DNN execution frameworks, such as TensorFlow, TVM, and MNN. However, these frameworks usually adopt fusion approaches based on certain patterns that are too restrictive to cover the diversity of operators and layer connections. Polyhedral-based loop fusion techniques, on the other hand, work on a low-level view of the computation without operator-level information, and can also miss potential fusion opportunities. To address this challenge, this paper proposes a novel and extensive loop fusion framework called DNNFusion. The basic idea of this work is to work at an operator view of DNNs, but expand fusion opportunities by developing a classification of both individual operators and their combinations. In addition, DNNFusion includes 1) a novel mathematical-property-based graph rewriting framework to reduce evaluation costs and facilitate subsequent operator fusion, 2) an integrated fusion plan generation that leverages the high-level analysis and accurate light-weight profiling, and 3) additional optimizations during fusion code generation. DNNFusion is extensively evaluated on 15 DNN models with varied types of tasks, model sizes, and layer counts. The evaluation results demonstrate that DNNFusion finds up to 8.8x higher fusion opportunities, outperforms four state-of-the-art DNN execution frameworks with 9.3x speedup. The memory requirement reduction and speedups can enable the execution of many of the target models on mobile devices and even make them part of a real-time application.