Tempo: Compiled Dynamic Deep Learning with Symbolic Dependence Graphs
Not on the map. Why?
Abstract (arXiv)
Deep learning (DL) algorithms are often defined in terms of temporal relationships: a tensor at one timestep may depend on tensors from earlier or later timesteps. Such dynamic dependencies (and corresponding dynamic tensor shapes) are difficult to express and optimize: while eager DL systems support such dynamism, they cannot apply compiler-based optimizations; graph-based systems require static tensor shapes, which forces users to pad tensors or break-up programs into multiple static graphs. We describe Tempo, a new DL system that combines the dynamism of eager execution with the whole-program optimizations of graph-based compilation. Tempo achieves this through a declarative programming model with recurrent tensors, which include explicit temporal dimensions. Temporal dimensions can be indexed using symbolic expressions to express dynamic dependencies on past and future tensors. Based on this, Tempo constructs a symbolic dependence graph, which concisely encodes dynamic dependencies between operators, and applies whole-program optimizations, such as algebraic simplifications, vectorization, tiling, and fusion. By tiling dynamic dependencies into static-size blocks, Tempo can also reuse existing static code-generators. It then uses a polyhedral model to find a feasible execution schedule, which includes memory management operations. We show that Tempo achieves a 7× speedup over JAX for Llama-3.2-3B decoding; for reinforcement learning algorithms, Tempo achieves a 54× speedup, with 16× lower peak memory usage.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 50 of its 98 references that are in the map. First posted on arXiv 9 Jan 2025.
Builds on
50 of 98 references.
References · 50
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2020Denoising Diffusion Probabilistic Modelscited by 20,851
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
38 more
- 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
- 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
- 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
- 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
- 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
- 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
- 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
- 2020Big Bird: Transformers for Longer Sequencescited by 1,673
- 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
- 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
- 2015Automatic differentiation in machine learning: a surveycited by 846
- 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
- 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
- 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
- 2016Temporal Convolutional Networks for Action Segmentation and Detectioncited by 504
- 2019Learning agile and dynamic motor skills for legged robotscited by 402
- 2021Flow Network based Generative Models for Non-Iterative Diverse Candidate Generationcited by 384
- 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
- 2019Reinforcement Learning in Healthcare: A Surveycited by 308
- 2014cuDNN: Efficient Primitives for Deep Learningcited by 240
- 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
- 2022Mega: Moving Average Equipped Gated Attentioncited by 178
- 2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 139
- 2022DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Realitycited by 136
- 2018Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractionscited by 84
- 2020Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learningcited by 84
- 2018SuperNeurons: Dynamic GPU Memory Management for Training Deep Neural Networkscited by 55
- 2018Tiramisu: A Polyhedral Compiler for Expressing Fast and Portable Codecited by 53
- 2019rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorchcited by 38
- 2021Podracer architectures for scalable Reinforcement Learningcited by 37
- 2021Reverb: A Framework For Experience Replaycited by 27
- 2019TensorFlow Eager: A Multi-Stage, Python-Embedded DSL for Machine Learningcited by 22
- 2020Nimble: Efficiently Compiling Dynamic Neural Networks for Model Inferencecited by 19
- 2021LazyTensor: combining eager execution with domain-specific compilerscited by 11
- 2018Dynamic Control Flow in Large-Scale Machine Learningcited by 8
- 2023GEAR: A GPU-Centric Experience Replay System for Large Reinforcement Learning Modelscited by 5
Built on by
Citing papers per year: 2026: 1.
1 paper.