# Salus: Fine-Grained GPU Sharing Primitives for Deep Learning Applications

arXiv: https://arxiv.org/abs/1902.04610 · HTML: https://mlsanity.com/paper/1902.04610/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Feb 2019.

## Built on by (most cited first, 17 of 17)

- [Sustainable AI: Environmental Implications, Challenges and Opportunities](https://mlsanity.com/paper/2111.00364/) (2021)
- [Characterization and Prediction of Deep Learning Workloads in Large-Scale GPU Datacenters](https://mlsanity.com/paper/2109.01313/) (2021)
- [Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooning](https://mlsanity.com/paper/2505.04021/) (2025)
- [Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Survey](https://mlsanity.com/paper/2404.06114/) (2024)
- [PipeFill: Using GPUs During Bubbles in Pipeline-parallel LLM Training](https://mlsanity.com/paper/2410.07192/) (2024)
- [ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Caching](https://mlsanity.com/paper/2509.16857/) (2025)
- [Doing More by Doing Less: How Structured Partial Backpropagation Improves Deep Learning Clusters](https://mlsanity.com/paper/2111.10672/) (2021)
- [Resource Allocation and Workload Scheduling for Large-Scale Distributed Deep Learning: A Survey](https://mlsanity.com/paper/2406.08115/) (2024)
- [A Study on the Intersection of GPU Utilization and CNN Inference](https://mlsanity.com/paper/2212.07936/) (2022)
- [MIGPerf: A Comprehensive Benchmark for Deep Learning Training and Inference Workloads on Multi-Instance GPUs](https://mlsanity.com/paper/2301.00407/) (2023)
- [Scheduling Optimization Techniques for Neural Network Training](https://mlsanity.com/paper/2110.00929/) (2021)
- [Automatic and Efficient Customization of Neural Networks for ML Applications](https://mlsanity.com/paper/2310.04685/) (2023)
- [Towards providing reliable job completion time predictions using PCS](https://mlsanity.com/paper/2401.10354/) (2024)
- [Hydro: Adaptive Query Processing of ML Queries](https://mlsanity.com/paper/2403.14902/) (2024)
- [Legilimens: Performant Video Analytics on the System-on-Chip Edge](https://mlsanity.com/paper/2504.21136/) (2025)
- [PlexRL: Cluster-Level Orchestration of Serviceized LLM Execution for RLVR](https://mlsanity.com/paper/2605.20863/) (2026)
- [AIBuildAI-2.5: Efficient Autonomous AI Model Development Through LLM-Guided Tree Search](https://mlsanity.com/paper/2609.25047/) (2026)

## Abstract (arXiv)

GPU computing is becoming increasingly more popular with the proliferation of deep learning (DL) applications. However, unlike traditional resources such as CPU or the network, modern GPUs do not natively support fine-grained sharing primitives. Consequently, implementing common policies such as time sharing and preemption are expensive. Worse, when a DL application cannot completely use a GPU's resources, the GPU cannot be efficiently shared between multiple applications, leading to GPU underutilization. We present Salus to enable two GPU sharing primitives: fast job switching and memory sharing, in order to achieve fine-grained GPU sharing among multiple DL applications. Salus implements an efficient, consolidated execution service that exposes the GPU to different DL applications, and enforces fine-grained sharing by performing iteration scheduling and addressing associated memory management issues. We show that these primitives can then be used to implement flexible sharing policies such as fairness, prioritization, and packing for various use cases. Our integration of Salus with TensorFlow and evaluation on popular DL jobs show that Salus can improve the average completion time of DL training jobs by 3.19×, GPU utilization for hyper-parameter tuning by 2.38×, and GPU utilization of DL inference applications by 42× over not sharing the GPU and 7× over NVIDIA MPS with small overhead.