mlsanityData to 7 Oct 2026

LazyTensor: combining eager execution with domain-specific compilers

arXiv 2102.13267 ↗First posted 26 Feb 2021cs.PL cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Domain-specific optimizing compilers have demonstrated significant performance and portability benefits, but require programs to be represented in their specialized IRs. Existing frontends to these compilers suffer from the "language subset problem" where some host language features are unsupported in the subset of the user's program that interacts with the domain-specific compiler. By contrast, define-by-run ML frameworks-colloquially called "eager" mode-are popular due to their ease of use and expressivity, where the full power of the host programming language can be used. LazyTensor is a technique to target domain specific compilers without sacrificing define-by-run ergonomics. Initially developed to support PyTorch on Cloud TPUs, the technique, along with a substantially shared implementation, has been used by Swift for TensorFlow across CPUs, GPUs, and TPUs, demonstrating the generality of the approach across (1) Tensor implementations, (2) hardware accelerators, and (3) programming languages.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 21 of its 34 references that are in the map. First posted on arXiv 26 Feb 2021.

Builds on

21 of 34 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2017Mask R-CNNcited by 7,259
  7. 2016Pointer Sentinel Mixture Modelscited by 2,992
  8. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  9. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  10. 2017Mixed Precision Trainingcited by 1,135
  11. 2014Caffe: Convolutional Architecture for Fast Feature Embeddingcited by 287
  12. 2014cuDNN: Efficient Primitives for Deep Learningcited by 240
9 more
  1. 2018Glow: Graph Lowering Compiler Techniques for Neural Networkscited by 73
  2. 2016Theano: A Python framework for fast computation of mathematical expressionscited by 64
  3. 2019TensorFlow Eager: A Multi-Stage, Python-Embedded DSL for Machine Learningcited by 22
  4. 2019Chainer: A Deep Learning Framework for Accelerating the Research Cyclecited by 22
  5. 2017DyNet: The Dynamic Neural Network Toolkitcited by 21
  6. 2018AutoGraph: Imperative-style Coding with Graph-based Performancecited by 9
  7. 2018Taichi: An Open-Source Computer Graphics Librarycited by 3
  8. 2018Automatic Full Compilation of Julia Programs and ML Models to Cloud TPUscited by 3
  9. 2018Learning to Discover, Ground and Use Words with Segmental Neural Language Modelscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 3; 2023: 3; 2024: 0; 2025: 3; 2026: 1.

11 papers.

  1. 2021Torch.fx: Practical Program Capture and Transformation for Deep Learning in Pythoncited by 38
  2. 2023Slapo: A Schedule Language for Progressive Optimization of Large Deep Learning Model Trainingcited by 11
  3. 2022Revisiting and Advancing Chinese Natural Language Understanding with Accelerated Heterogeneous Knowledge Pre-trainingcited by 6
  4. 2022Memory Planning for Deep Neural Networkscited by 3
  5. 2025PyGraph: Robust Compiler Support for CUDA Graphs in PyTorchcited by 3
  6. 2022Terra: Imperative-Symbolic Co-Execution of Imperative Deep Learning Programscited by 1
  7. 2023RAF: Holistic Compilation for Deep Learning Model Trainingcited by 1
  8. 2023ACRoBat: Optimizing Auto-batching of Dynamic Deep Learning at Compile Timecited by 1
  9. 2025Tempo: Compiled Dynamic Deep Learning with Symbolic Dependence Graphscited by 1
  10. 2025Speculative Automated Refactoring of Imperative Deep Learning Programs to Graph Executioncited by 0
  11. 2026DVM: A Bytecode Virtual Machine Approach for Dynamic Tensor Computationcited by 0