# Relay: A New IR for Machine Learning Frameworks

arXiv: https://arxiv.org/abs/1810.00952 · HTML: https://mlsanity.com/paper/1810.00952/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Sep 2018.

## Built on by (most cited first, 24 of 24)

- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [GSPMD: General and Scalable Parallelization for ML Computation Graphs](https://mlsanity.com/paper/2105.04663/) (2021)
- [SparseTIR: Composable Abstractions for Sparse Compilation in Deep Learning](https://mlsanity.com/paper/2207.04606/) (2022)
- [Zero Bubble Pipeline Parallelism](https://mlsanity.com/paper/2401.10241/) (2023)
- [Relax: Composable Abstractions for End-to-End Dynamic Machine Learning](https://mlsanity.com/paper/2311.02103/) (2023)
- [UNIT: Unifying Tensorized Instruction Compilation](https://mlsanity.com/paper/2101.08458/) (2021)
- [ImpNet: Imperceptible and blackbox-undetectable backdoors in compiled neural networks](https://mlsanity.com/paper/2210.00108/) (2022)
- [A Highly Configurable Hardware/Software Stack for DNN Inference Acceleration](https://mlsanity.com/paper/2111.15024/) (2021)
- [SplitSR: An End-to-End Approach to Super-Resolution on Mobile Devices](https://mlsanity.com/paper/2101.07996/) (2021)
- [Memory Planning for Deep Neural Networks](https://mlsanity.com/paper/2203.00448/) (2022)
- [Collage: Seamless Integration of Deep Learning Backends with Automatic Placement](https://mlsanity.com/paper/2111.00655/) (2021)
- [CPrune: Compiler-Informed Model Pruning for Efficient Target-Aware DNN Execution](https://mlsanity.com/paper/2207.01260/) (2022)
- [Understanding Cache Boundness of ML Operators on ARM Processors](https://mlsanity.com/paper/2102.00932/) (2021)
- [Joint Program and Layout Transformations to enable Convolutional Operators on Specialized Hardware based on Constraint Programming](https://mlsanity.com/paper/2104.04731/) (2021)
- [Bifrost: End-to-End Evaluation and Optimization of Reconfigurable DNN Accelerators](https://mlsanity.com/paper/2204.12418/) (2022)
- [AGO: Boosting Mobile AI Inference Performance by Removing Constraints on Graph Optimization](https://mlsanity.com/paper/2212.01005/) (2022)
- [DIPPM: a Deep Learning Inference Performance Predictive Model using Graph Neural Networks](https://mlsanity.com/paper/2303.11733/) (2023)
- [A Compilation Flow for the Generation of CNN Inference Accelerators on FPGAs](https://mlsanity.com/paper/2203.04015/) (2022)
- [Faith: An Efficient Framework for Transformer Verification on GPUs](https://mlsanity.com/paper/2209.12708/) (2022)
- [ALT: Boosting Deep Learning Performance by Breaking the Wall between Graph and Operator Level Optimizations](https://mlsanity.com/paper/2210.12415/) (2022)
- [SparseOptimizer: Sparsify Language Models through Moreau-Yosida Regularization and Accelerate via Compiler Co-design](https://mlsanity.com/paper/2306.15656/) (2023)
- [A Survey on Design Methodologies for Accelerating Deep Learning on Heterogeneous Architectures](https://mlsanity.com/paper/2311.17815/) (2023)
- [LightCode: Compiling LLM Inference for Photonic-Electronic Systems](https://mlsanity.com/paper/2509.16443/) (2025)
- [Accelerated Execution of Bayesian Neural Networks using a Single Probabilistic Forward Pass and Code Generation](https://mlsanity.com/paper/2511.23440/) (2025)

## Abstract (arXiv)

Machine learning powers diverse services in industry including search, translation, recommendation systems, and security. The scale and importance of these models require that they be efficient, expressive, and portable across an array of heterogeneous hardware devices. These constraints are often at odds; in order to better accommodate them we propose a new high-level intermediate representation (IR) called Relay. Relay is being designed as a purely-functional, statically-typed language with the goal of balancing efficient compilation, expressiveness, and portability. We discuss the goals of Relay and highlight its important design constraints. Our prototype is part of the open source NNVM compiler framework, which powers Amazon's deep learning framework MxNet.