# MuxFlow: Efficient and Safe GPU Sharing in Large-Scale Production Deep Learning Clusters

arXiv: https://arxiv.org/abs/2303.13803 · HTML: https://mlsanity.com/paper/2303.13803/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Mar 2023.

## Built on by (most cited first, 7 of 7)

- [MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving](https://mlsanity.com/paper/2404.02015/) (2024)
- [Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooning](https://mlsanity.com/paper/2505.04021/) (2025)
- [FlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guarantees](https://mlsanity.com/paper/2402.18789/) (2024)
- [PolyThrottle: Energy-efficient Neural Network Inference on Edge Devices](https://mlsanity.com/paper/2310.19991/) (2023)
- [Resource Allocation and Workload Scheduling for Large-Scale Distributed Deep Learning: A Survey](https://mlsanity.com/paper/2406.08115/) (2024)
- [Tesserae: Scalable Placement Policies for Deep Learning Workloads](https://mlsanity.com/paper/2508.04953/) (2025)
- [Evaluating Kubernetes Performance for GenAI Inference: From Automatic Speech Recognition to LLM Summarization](https://mlsanity.com/paper/2602.04900/) (2026)

## Abstract (arXiv)

Large-scale GPU clusters are widely-used to speed up both latency-critical (online) and best-effort (offline) deep learning (DL) workloads. However, most DL clusters either dedicate each GPU to one workload or share workloads in time, leading to very low GPU resource utilization. We present MuxFlow, the first production cluster system that supports efficient and safe space-sharing for DL workloads. NVIDIA MPS provides an opportunity to share multiple workloads in space on widely-deployed NVIDIA GPUs, but it cannot guarantee the performance and safety of online workloads. MuxFlow introduces a two-level protection mechanism for memory and computation to guarantee the performance of online workloads. Based on our practical error analysis, we design a mixed error-handling mechanism to guarantee the safety of online workloads. MuxFlow further proposes dynamic streaming multiprocessor (SM) allocation and matching-based scheduling to improve the efficiency of offline workloads. MuxFlow has been deployed at CompanyX's clusters with more than 20,000 GPUs. The deployment results indicate that MuxFlow substantially improves the GPU utilization from 26% to 76%, SM activity from 16% to 33%, and GPU memory from 42% to 48%.