# Nimble: Efficiently Compiling Dynamic Neural Networks for Model Inference

arXiv: https://arxiv.org/abs/2006.03031 · HTML: https://mlsanity.com/paper/2006.03031/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jun 2020.

## Built on by (most cited first, 19 of 19)

- [TorchSparse++: Efficient Training and Inference Framework for Sparse Convolution on GPUs](https://mlsanity.com/paper/2311.12862/) (2023)
- [Hidet: Task-Mapping Programming Paradigm for Deep Learning Tensor Programs](https://mlsanity.com/paper/2210.09603/) (2022)
- [The CoRa Tensor Compiler: Compilation for Ragged Tensors with Minimal Padding](https://mlsanity.com/paper/2110.10221/) (2021)
- [Transformer-Lite: High-efficiency Deployment of Large Language Models on Mobile Phone GPUs](https://mlsanity.com/paper/2403.20041/) (2024)
- [ShiftAddViT: Mixture of Multiplication Primitives Towards Efficient Vision Transformer](https://mlsanity.com/paper/2306.06446/) (2023)
- [Relax: Composable Abstractions for End-to-End Dynamic Machine Learning](https://mlsanity.com/paper/2311.02103/) (2023)
- [SmartMem: Layout Transformation Elimination and Adaptation for Efficient DNN Execution on Mobile](https://mlsanity.com/paper/2404.13528/) (2024)
- [DyCL: Dynamic Neural Network Compilation Via Program Rewriting and Graph Optimization](https://mlsanity.com/paper/2307.04963/) (2023)
- [Bring Your Own Codegen to Deep Learning Compiler](https://mlsanity.com/paper/2105.03215/) (2021)
- [SoD2: Statically Optimizing Dynamic Deep Neural Network](https://mlsanity.com/paper/2403.00176/) (2024)
- [Memory Planning for Deep Neural Networks](https://mlsanity.com/paper/2203.00448/) (2022)
- [PyGraph: Robust Compiler Support for CUDA Graphs in PyTorch](https://mlsanity.com/paper/2503.19779/) (2025)
- [Tilus: A Tile-Level GPGPU Programming Language for Low-Precision Computation](https://mlsanity.com/paper/2504.12984/) (2025)
- [How to Reach Real-Time AI on Consumer Devices? Solutions for Programmable and Custom Architectures](https://mlsanity.com/paper/2106.15021/) (2021)
- [Tempo: Compiled Dynamic Deep Learning with Symbolic Dependence Graphs](https://mlsanity.com/paper/2501.05408/) (2025)
- [ALT: Boosting Deep Learning Performance by Breaking the Wall between Graph and Operator Level Optimizations](https://mlsanity.com/paper/2210.12415/) (2022)
- [FTuner: A Fast Dynamic Shape Tensors Program Auto-Tuner for Deep Learning Compilers](https://mlsanity.com/paper/2407.21418/) (2024)
- [Parallax: Runtime Parallelization for Operator Fallbacks in Heterogeneous Edge Systems](https://mlsanity.com/paper/2512.11532/) (2025)
- [DVM: A Bytecode Virtual Machine Approach for Dynamic Tensor Computation](https://mlsanity.com/paper/2603.24239/) (2026)

## Abstract (arXiv)

Modern deep neural networks increasingly make use of features such as dynamic control flow, data structures and dynamic tensor shapes. Existing deep learning systems focus on optimizing and executing static neural networks which assume a pre-determined model architecture and input data shapes--assumptions which are violated by dynamic neural networks. Therefore, executing dynamic models with deep learning systems is currently both inflexible and sub-optimal, if not impossible. Optimizing dynamic neural networks is more challenging than static neural networks; optimizations must consider all possible execution paths and tensor shapes. This paper proposes Nimble, a high-performance and flexible system to optimize, compile, and execute dynamic neural networks on multiple platforms. Nimble handles model dynamism by introducing a dynamic type system, a set of dynamism-oriented optimizations, and a light-weight virtual machine runtime. Our evaluation demonstrates that Nimble outperforms state-of-the-art deep learning frameworks and runtime systems for dynamic neural networks by up to 20x on hardware platforms including Intel CPUs, ARM CPUs, and Nvidia GPUs.