# Serving DNN Models with Multi-Instance GPUs: A Case of the Reconfigurable Machine Scheduling Problem

arXiv: https://arxiv.org/abs/2109.11067 · HTML: https://mlsanity.com/paper/2109.11067/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 38 references that are in the map.
- First posted on arXiv 18 Sep 2021.

## Builds on (most cited first, 8 of 8)

- [Clipper: A Low-Latency Online Prediction Serving System](https://mlsanity.com/paper/1612.03079/) (2016)
- [Serving DNNs like Clockwork: Performance Predictability from the Bottom Up](https://mlsanity.com/paper/2006.02464/) (2020)
- [TensorFlow-Serving: Flexible, High-Performance ML Serving](https://mlsanity.com/paper/1712.06139/) (2017)
- [Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning](https://mlsanity.com/paper/2008.12260/) (2020)
- [Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloads](https://mlsanity.com/paper/2008.09213/) (2020)
- [Themis: Fair and Efficient GPU Cluster Scheduling](https://mlsanity.com/paper/1907.01484/) (2019)
- [Accelerating Deep Learning Inference via Learned Caches](https://mlsanity.com/paper/2101.07344/) (2021)
- [Approximation Algorithms for Scheduling with Class Constraints](https://mlsanity.com/paper/1909.11970/) (2019)

## Built on by (most cited first, 7 of 7)

- [MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving](https://mlsanity.com/paper/2404.02015/) (2024)
- [HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-location](https://mlsanity.com/paper/2501.14808/) (2025)
- [LithOS: An Operating System for Efficient Machine Learning on GPUs](https://mlsanity.com/paper/2504.15465/) (2025)
- [Deep Learning Workload Scheduling in GPU Datacenters: Taxonomy, Challenges and Vision](https://mlsanity.com/paper/2205.11913/) (2022)
- [MIGPerf: A Comprehensive Benchmark for Deep Learning Training and Inference Workloads on Multi-Instance GPUs](https://mlsanity.com/paper/2301.00407/) (2023)
- [A Survey of Serverless Machine Learning Model Inference](https://mlsanity.com/paper/2311.13587/) (2023)
- [Energy-Aware Scheduling for Serverless LLM Serving on Shared GPUs](https://mlsanity.com/paper/2606.30391/) (2026)

## Abstract (arXiv)

Multi-Instance GPU (MIG) is a new feature introduced by NVIDIA A100 GPUs that partitions one physical GPU into multiple GPU instances. With MIG, A100 can be the most cost-efficient GPU ever for serving Deep Neural Networks (DNNs). However, discovering the most efficient GPU partitions is challenging. The underlying problem is NP-hard; moreover, it is a new abstract problem, which we define as the Reconfigurable Machine Scheduling Problem (RMS). This paper studies serving DNNs with MIG, a new case of RMS. We further propose a solution, MIG-serving. MIG- serving is an algorithm pipeline that blends a variety of newly designed algorithms and customized classic algorithms, including a heuristic greedy algorithm, Genetic Algorithm (GA), and Monte Carlo Tree Search algorithm (MCTS). We implement MIG-serving on Kubernetes. Our experiments show that compared to using A100 as-is, MIG-serving can save up to 40% of GPUs while providing the same throughput.