# Characterizing Concurrency Mechanisms for NVIDIA GPUs under Deep Learning Workloads

arXiv: https://arxiv.org/abs/2110.00459 · HTML: https://mlsanity.com/paper/2110.00459/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 29 references that are in the map.
- First posted on arXiv 1 Oct 2021.

## Builds on (most cited first, 6 of 6)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [MLPerf Inference Benchmark](https://mlsanity.com/paper/1911.02549/) (2019)
- [Nimble: Lightweight and Parallel GPU Task Scheduling for Deep Learning](https://mlsanity.com/paper/2012.02732/) (2020)
- [Dynamic Space-Time Scheduling for GPU Inference](https://mlsanity.com/paper/1901.00041/) (2018)

## Built on by (most cited first, 2 of 2)

- [Aryl: An Elastic Cluster Scheduler for Deep Learning](https://mlsanity.com/paper/2202.07896/) (2022)
- [Strait: Perceiving Priority and Interference in ML Inference Serving](https://mlsanity.com/paper/2604.28175/) (2026)

## Abstract (arXiv)

We investigate the performance of the concurrency mechanisms available on NVIDIA's new Ampere GPU microarchitecture under deep learning training and inference workloads. In contrast to previous studies that treat the GPU as a black box, we examine scheduling at the microarchitectural level. We find that the lack of fine-grained preemption mechanisms, robust task prioritization options, and contention-aware thread block placement policies limits the effectiveness of NVIDIA's concurrency mechanisms. In summary, the sequential nature of deep learning workloads and their fluctuating resource requirements and kernel runtimes make executing such workloads while maintaining consistently high utilization and low, predictable turnaround times difficult on current NVIDIA hardware.