# LazyTensor: combining eager execution with domain-specific compilers

arXiv: https://arxiv.org/abs/2102.13267 · HTML: https://mlsanity.com/paper/2102.13267/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 34 references that are in the map.
- First posted on arXiv 26 Feb 2021.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [Caffe: Convolutional Architecture for Fast Feature Embedding](https://mlsanity.com/paper/1408.5093/) (2014)
- [cuDNN: Efficient Primitives for Deep Learning](https://mlsanity.com/paper/1410.0759/) (2014)
- [Glow: Graph Lowering Compiler Techniques for Neural Networks](https://mlsanity.com/paper/1805.00907/) (2018)
- [Theano: A Python framework for fast computation of mathematical expressions](https://mlsanity.com/paper/1605.02688/) (2016)
- [TensorFlow Eager: A Multi-Stage, Python-Embedded DSL for Machine Learning](https://mlsanity.com/paper/1903.01855/) (2019)
- [Chainer: A Deep Learning Framework for Accelerating the Research Cycle](https://mlsanity.com/paper/1908.00213/) (2019)
- [DyNet: The Dynamic Neural Network Toolkit](https://mlsanity.com/paper/1701.03980/) (2017)
- [AutoGraph: Imperative-style Coding with Graph-based Performance](https://mlsanity.com/paper/1810.08061/) (2018)
- [Taichi: An Open-Source Computer Graphics Library](https://mlsanity.com/paper/1804.09293/) (2018)
- [Automatic Full Compilation of Julia Programs and ML Models to Cloud TPUs](https://mlsanity.com/paper/1810.09868/) (2018)
- [Learning to Discover, Ground and Use Words with Segmental Neural Language Models](https://mlsanity.com/paper/1811.09353/) (2018)

## Built on by (most cited first, 11 of 11)

- [Torch.fx: Practical Program Capture and Transformation for Deep Learning in Python](https://mlsanity.com/paper/2112.08429/) (2021)
- [Slapo: A Schedule Language for Progressive Optimization of Large Deep Learning Model Training](https://mlsanity.com/paper/2302.08005/) (2023)
- [Revisiting and Advancing Chinese Natural Language Understanding with Accelerated Heterogeneous Knowledge Pre-training](https://mlsanity.com/paper/2210.05287/) (2022)
- [Memory Planning for Deep Neural Networks](https://mlsanity.com/paper/2203.00448/) (2022)
- [PyGraph: Robust Compiler Support for CUDA Graphs in PyTorch](https://mlsanity.com/paper/2503.19779/) (2025)
- [Terra: Imperative-Symbolic Co-Execution of Imperative Deep Learning Programs](https://mlsanity.com/paper/2201.09210/) (2022)
- [RAF: Holistic Compilation for Deep Learning Model Training](https://mlsanity.com/paper/2303.04759/) (2023)
- [ACRoBat: Optimizing Auto-batching of Dynamic Deep Learning at Compile Time](https://mlsanity.com/paper/2305.10611/) (2023)
- [Tempo: Compiled Dynamic Deep Learning with Symbolic Dependence Graphs](https://mlsanity.com/paper/2501.05408/) (2025)
- [Speculative Automated Refactoring of Imperative Deep Learning Programs to Graph Execution](https://mlsanity.com/paper/2504.05424/) (2025)
- [DVM: A Bytecode Virtual Machine Approach for Dynamic Tensor Computation](https://mlsanity.com/paper/2603.24239/) (2026)

## Abstract (arXiv)

Domain-specific optimizing compilers have demonstrated significant performance and portability benefits, but require programs to be represented in their specialized IRs. Existing frontends to these compilers suffer from the "language subset problem" where some host language features are unsupported in the subset of the user's program that interacts with the domain-specific compiler. By contrast, define-by-run ML frameworks-colloquially called "eager" mode-are popular due to their ease of use and expressivity, where the full power of the host programming language can be used. LazyTensor is a technique to target domain specific compilers without sacrificing define-by-run ergonomics. Initially developed to support PyTorch on Cloud TPUs, the technique, along with a substantially shared implementation, has been used by Swift for TensorFlow across CPUs, GPUs, and TPUs, demonstrating the generality of the approach across (1) Tensor implementations, (2) hardware accelerators, and (3) programming languages.