OLLIE: Derivation-based Tensor Program Optimizer
Not on the map. Why?
Abstract (arXiv)
Boosting the runtime performance of deep neural networks (DNNs) is critical due to their wide adoption in real-world tasks. Existing approaches to optimizing the tensor algebra expression of a DNN only consider expressions representable by a fixed set of predefined operators, missing possible optimization opportunities between general expressions. We propose OLLIE, the first derivation-based tensor program optimizer. OLLIE optimizes tensor programs by leveraging transformations between general tensor algebra expressions, enabling a significantly larger expression search space that includes those supported by prior work as special cases. OLLIE uses a hybrid derivation-based optimizer that effectively combines explorative and guided derivations to quickly discover highly optimized expressions. Evaluation on seven DNNs shows that OLLIE can outperform existing optimizers by up to 2.73× (1.46× on average) on an A100 GPU and up to 2.68× (1.51×) on a V100 GPU, respectively.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 23 of its 40 references that are in the map. First posted on arXiv 2 Aug 2022.
Builds on
23 of 40 references.
References · 23
- 2017Attention Is All You Needcited by 60,361
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2017Mask R-CNNcited by 7,277
- 2020Longformer: The Long-Document Transformercited by 3,413
- 2015Fast R-CNNcited by 2,933
- 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,054
- 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
- 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
- 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 860
- 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
- 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 618
- 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 451
11 more
- 2019A Survey of Deep Learning Techniques for Autonomous Drivingcited by 403
- 2018CSRNet: Dilated Convolutional Neural Networks for Understanding the Highly Congested Scenescited by 267
- 2014cuDNN: Efficient Primitives for Deep Learningcited by 245
- 2017Large Kernel Matters -- Improve Semantic Segmentation by Global Convolutional Networkcited by 208
- 2020Ansor: Generating High-Performance Tensor Programs for Deep Learningcited by 201
- 2018Learning to Optimize Tensor Programscited by 140
- 2015Fast Algorithms for Convolutional Neural Networkscited by 102
- 2021DNNFusion: Accelerating Deep Neural Networks Execution with Advanced Operator Fusioncited by 57
- 2018Tiramisu: A Polyhedral Compiler for Expressing Fast and Portable Codecited by 53
- 2020Computing Systems for Autonomous Driving: State-of-the-Art and Challengescited by 47
- 2017Parallel Multi Channel Convolution using General Matrix Multiplicationcited by 30
Built on by
Citing papers per year: 2022: 1.
1 paper.