# AGO: Boosting Mobile AI Inference Performance by Removing Constraints on Graph Optimization

arXiv: https://arxiv.org/abs/2212.01005 · HTML: https://mlsanity.com/paper/2212.01005/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 52 references that are in the map.
- First posted on arXiv 2 Dec 2022.

## Builds on (most cited first, 24 of 24)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer](https://mlsanity.com/paper/2110.02178/) (2021)
- [Ansor: Generating High-Performance Tensor Programs for Deep Learning](https://mlsanity.com/paper/2006.06762/) (2020)
- [Learning to Optimize Tensor Programs](https://mlsanity.com/paper/1805.08166/) (2018)
- [Well-Read Students Learn Better: On the Importance of Pre-training Compact Models](https://mlsanity.com/paper/1908.08962/) (2019)
- [SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloud](https://mlsanity.com/paper/2008.06402/) (2020)
- [Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions](https://mlsanity.com/paper/1802.04730/) (2018)
- [Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics](https://mlsanity.com/paper/2110.01518/) (2021)
- [MNN: A Universal and Efficient Inference Engine](https://mlsanity.com/paper/2002.12418/) (2020)
- [DNNFusion: Accelerating Deep Neural Networks Execution with Advanced Operator Fusion](https://mlsanity.com/paper/2108.13342/) (2021)
- [Tiramisu: A Polyhedral Compiler for Expressing Fast and Portable Code](https://mlsanity.com/paper/1804.10694/) (2018)
- [A Deep Learning Based Cost Model for Automatic Code Optimization](https://mlsanity.com/paper/2104.04955/) (2021)
- [Ordering Chaos: Memory-Aware Scheduling of Irregularly Wired Neural Networks for Edge Devices](https://mlsanity.com/paper/2003.02369/) (2020)
- [Equality Saturation for Tensor Graph Superoptimization](https://mlsanity.com/paper/2101.01332/) (2021)
- [Relay: A New IR for Machine Learning Frameworks](https://mlsanity.com/paper/1810.00952/) (2018)
- [Walle: An End-to-End, General-Purpose, and Large-Scale Production System for Device-Cloud Collaborative Machine Learning](https://mlsanity.com/paper/2205.14833/) (2022)
- [IOS: Inter-Operator Scheduler for CNN Acceleration](https://mlsanity.com/paper/2011.01302/) (2020)
- [Bolt: Bridging the Gap between Auto-tuners and Hardware-native Performance](https://mlsanity.com/paper/2110.15238/) (2021)
- [FusionStitching: Boosting Memory Intensive Computations for Deep Learning Workloads](https://mlsanity.com/paper/2009.10924/) (2020)
- [Tuna: A Static Analysis Approach to Optimizing Deep Neural Networks](https://mlsanity.com/paper/2104.14641/) (2021)

## Built on by (most cited first, 1 of 1)

- [Accelerating End-Cloud Collaborative Inference via Near Bubble-free Pipeline Optimization](https://mlsanity.com/paper/2501.12388/) (2024)

## Abstract (arXiv)

Traditional deep learning compilers rely on heuristics for subgraph generation, which impose extra constraints on graph optimization, e.g., each subgraph can only contain at most one complex operator. In this paper, we propose AGO, a framework for graph optimization with arbitrary structures to boost the inference performance of deep models by removing such constraints. To create new optimization opportunities for complicated subgraphs, we propose intensive operator fusion, which can effectively stitch multiple complex operators together for better performance. Further, we design a graph partitioning scheme that allows an arbitrary structure for each subgraph while guaranteeing the acyclic property among all generated subgraphs. Additionally, to enable efficient performance tuning on complicated subgraphs, we devise a novel divide-and-conquer tuning mechanism to orchestrate different system components. Through extensive experiments on various neural networks and mobile devices, we show that our system can improve the inference performance by up to 3.3x when compared with state-of-the-art deep compilers.