# Intel nGraph: An Intermediate Representation, Compiler, and Executor for Deep Learning

arXiv: https://arxiv.org/abs/1801.08058 · HTML: https://mlsanity.com/paper/1801.08058/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jan 2018.

## Built on by (most cited first, 23 of 23)

- [TLP: A Deep Learning-based Cost Model for Tensor Program Tuning](https://mlsanity.com/paper/2211.03578/) (2022)
- [GraphQ IR: Unifying the Semantic Parsing of Graph Query Languages with One Intermediate Representation](https://mlsanity.com/paper/2205.12078/) (2022)
- [Learning to Reverse DNNs from AI Programs Automatically](https://mlsanity.com/paper/2205.10364/) (2022)
- [DyCL: Dynamic Neural Network Compilation Via Program Rewriting and Graph Optimization](https://mlsanity.com/paper/2307.04963/) (2023)
- [Neural Architecture Search as Program Transformation Exploration](https://mlsanity.com/paper/2102.06599/) (2021)
- [Enabling Homomorphically Encrypted Inference for Large DNN Models](https://mlsanity.com/paper/2103.16139/) (2021)
- [Quantune: Post-training Quantization of Convolutional Neural Networks using Extreme Gradient Boosting for Fast Deployment](https://mlsanity.com/paper/2202.05048/) (2022)
- [A Data-Centric Optimization Framework for Machine Learning](https://mlsanity.com/paper/2110.10802/) (2021)
- [Benchmarking the Linear Algebra Awareness of TensorFlow and PyTorch](https://mlsanity.com/paper/2202.09888/) (2022)
- [CIM-MLC: A Multi-level Compilation Stack for Computing-In-Memory Accelerators](https://mlsanity.com/paper/2401.12428/) (2024)
- [NNStreamer: Efficient and Agile Development of On-Device AI Systems](https://mlsanity.com/paper/2101.06371/) (2021)
- [Collage: Seamless Integration of Deep Learning Backends with Automatic Placement](https://mlsanity.com/paper/2111.00655/) (2021)
- [Compiler Toolchains for Deep Learning Workloads on Embedded Platforms](https://mlsanity.com/paper/2104.04576/) (2021)
- [Joint Program and Layout Transformations to enable Convolutional Operators on Specialized Hardware based on Constraint Programming](https://mlsanity.com/paper/2104.04731/) (2021)
- [LoopTune: Optimizing Tensor Computations with Reinforcement Learning](https://mlsanity.com/paper/2309.01825/) (2023)
- [Proteus: Preserving Model Confidentiality during Graph Optimizations](https://mlsanity.com/paper/2404.12512/) (2024)
- [vMCU: Coordinated Memory Management and Kernel Optimization for DNN Inference on MCUs](https://mlsanity.com/paper/2406.06542/) (2024)
- [FamilySeer: Towards Optimized Tensor Codes by Exploiting Computation Subgraph Similarity](https://mlsanity.com/paper/2201.00194/) (2022)
- [ALT: Boosting Deep Learning Performance by Breaking the Wall between Graph and Operator Level Optimizations](https://mlsanity.com/paper/2210.12415/) (2022)
- [CMLCompiler: A Unified Compiler for Classical Machine Learning](https://mlsanity.com/paper/2301.13441/) (2023)
- [GEVO-ML: Optimizing Machine Learning Code with Evolutionary Computation](https://mlsanity.com/paper/2310.10211/) (2023)
- [PolyTOPS: Reconfigurable and Flexible Polyhedral Scheduler](https://mlsanity.com/paper/2401.06665/) (2024)
- [Grounding Deep Learning Framework Testing in Data Flow](https://mlsanity.com/paper/2610.15808/) (2026)

## Abstract (arXiv)

The Deep Learning (DL) community sees many novel topologies published each year. Achieving high performance on each new topology remains challenging, as each requires some level of manual effort. This issue is compounded by the proliferation of frameworks and hardware platforms. The current approach, which we call "direct optimization", requires deep changes within each framework to improve the training performance for each hardware backend (CPUs, GPUs, FPGAs, ASICs) and requires O(fp) effort; where f is the number of frameworks and p is the number of platforms. While optimized kernels for deep-learning primitives are provided via libraries like Intel Math Kernel Library for Deep Neural Networks (MKL-DNN), there are several compiler-inspired ways in which performance can be further optimized. Building on our experience creating neon (a fast deep learning library on GPUs), we developed Intel nGraph, a soon to be open-sourced C++ library to simplify the realization of optimized deep learning performance across frameworks and hardware platforms. Initially-supported frameworks include TensorFlow, MXNet, and Intel neon framework. Initial backends are Intel Architecture CPUs (CPU), the Intel(R) Nervana Neural Network Processor(R) (NNP), and NVIDIA GPUs. Currently supported compiler optimizations include efficient memory management and data layout abstraction. In this paper, we describe our overall architecture and its core components. In the future, we envision extending nGraph API support to a wider range of frameworks, hardware (including FPGAs and ASICs), and compiler optimizations (training versus inference optimizations, multi-node and multi-device scaling via efficient sub-graph partitioning, and HW-specific compounding of operations).