# Nimble: Lightweight and Parallel GPU Task Scheduling for Deep Learning

arXiv: https://arxiv.org/abs/2012.02732 · HTML: https://mlsanity.com/paper/2012.02732/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Dec 2020.

## Built on by (most cited first, 17 of 17)

- [A Fast Post-Training Pruning Framework for Transformers](https://mlsanity.com/paper/2204.09656/) (2022)
- [Pathways: Asynchronous Distributed Dataflow for ML](https://mlsanity.com/paper/2203.12533/) (2022)
- [Multi-model Machine Learning Inference Serving with GPU Spatial Partitioning](https://mlsanity.com/paper/2109.01611/) (2021)
- [PiPAD: Pipelined and Parallel Dynamic GNN Training on GPUs](https://mlsanity.com/paper/2301.00391/) (2023)
- [Yggdrasil: Bridging Dynamic Speculation and Static Runtime for Latency-Optimal Tree-Based LLM Decoding](https://mlsanity.com/paper/2512.23858/) (2025)
- [Characterizing Concurrency Mechanisms for NVIDIA GPUs under Deep Learning Workloads](https://mlsanity.com/paper/2110.00459/) (2021)
- [Baechi: Fast Device Placement of Machine Learning Graphs](https://mlsanity.com/paper/2301.08695/) (2023)
- [Terra: Imperative-Symbolic Co-Execution of Imperative Deep Learning Programs](https://mlsanity.com/paper/2201.09210/) (2022)
- [Optimal channel selection with discrete QCQP](https://mlsanity.com/paper/2202.12417/) (2022)
- [ED-Batch: Efficient Automatic Batching of Dynamic Neural Networks via Learned Finite State Machines](https://mlsanity.com/paper/2302.03851/) (2023)
- [A Differentiable Framework for End-to-End Learning of Hybrid Structured Compression](https://mlsanity.com/paper/2309.13077/) (2023)
- [Scheduling Optimization Techniques for Neural Network Training](https://mlsanity.com/paper/2110.00929/) (2021)
- [Opara: Exploiting Operator Parallelism for Expediting DNN Inference on GPUs](https://mlsanity.com/paper/2312.10351/) (2023)
- [Compass: A Decentralized Scheduler for Latency-Sensitive ML Workflows](https://mlsanity.com/paper/2402.17652/) (2024)
- [Convex Optimization for Alignment and Preference Learning on a Single GPU](https://mlsanity.com/paper/2605.23244/) (2026)
- [CTA-Pipelining: A Latency-Oriented Spatial Scaling Method for Multi-GPU Systems](https://mlsanity.com/paper/2607.07862/) (2026)
- [DuplexCadence: Exact State and Execution from a Speech Model's Declared Timelines](https://mlsanity.com/paper/2609.35115/) (2026)

## Abstract (arXiv)

Deep learning (DL) frameworks take advantage of GPUs to improve the speed of DL inference and training. Ideally, DL frameworks should be able to fully utilize the computation power of GPUs such that the running time depends on the amount of computation assigned to GPUs. Yet, we observe that in scheduling GPU tasks, existing DL frameworks suffer from inefficiencies such as large scheduling overhead and unnecessary serial execution. To this end, we propose Nimble, a DL execution engine that runs GPU tasks in parallel with minimal scheduling overhead. Nimble introduces a novel technique called ahead-of-time (AoT) scheduling. Here, the scheduling procedure finishes before executing the GPU kernel, thereby removing most of the scheduling overhead during run time. Furthermore, Nimble automatically parallelizes the execution of GPU tasks by exploiting multiple GPU streams in a single GPU. Evaluation on a variety of neural networks shows that compared to PyTorch, Nimble speeds up inference and training by up to 22.34× and 3.61×, respectively. Moreover, Nimble outperforms state-of-the-art inference systems, TensorRT and TVM, by up to 2.81× and 1.70×, respectively.