# MISO: Exploiting Multi-Instance GPU Capability on Multi-Tenant Systems for Machine Learning

arXiv: https://arxiv.org/abs/2207.11428 · HTML: https://mlsanity.com/paper/2207.11428/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jul 2022.

## Built on by (most cited first, 15 of 15)

- [From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference](https://mlsanity.com/paper/2310.03003/) (2023)
- [LithOS: An Operating System for Efficient Machine Learning on GPUs](https://mlsanity.com/paper/2504.15465/) (2025)
- [An Analysis of Collocation on GPUs for Deep Learning Training](https://mlsanity.com/paper/2209.06018/) (2022)
- [SCAR: Scheduling Multi-Model AI Workloads on Heterogeneous Multi-Chiplet Module Accelerators](https://mlsanity.com/paper/2405.00790/) (2024)
- [COMET: Towards Partical W4A4KV4 LLMs Serving](https://mlsanity.com/paper/2410.12168/) (2024)
- [Fair Resource Allocation in Virtualized O-RAN Platforms](https://mlsanity.com/paper/2402.11285/) (2024)
- [KAIROS: Building Cost-Efficient Machine Learning Inference Systems with Heterogeneous Cloud Resources](https://mlsanity.com/paper/2210.05889/) (2022)
- [Hierarchical Resource Partitioning on Modern GPUs: A Reinforcement Learning Approach](https://mlsanity.com/paper/2405.08754/) (2024)
- [DIPPM: a Deep Learning Inference Performance Predictive Model using Graph Neural Networks](https://mlsanity.com/paper/2303.11733/) (2023)
- [xMem: A CPU-Based Approach for Accurate Estimation of GPU Memory in Deep Learning Training Workloads](https://mlsanity.com/paper/2510.21048/) (2025)
- [PREBA: A Hardware/Software Co-Design for Multi-Instance GPU based AI Inference Servers](https://mlsanity.com/paper/2411.19114/) (2024)
- [Survey of HPC in US Research Institutions](https://mlsanity.com/paper/2506.19019/) (2025)
- [Predictable LLM Serving on GPU Clusters](https://mlsanity.com/paper/2508.20274/) (2025)
- [VUDA: Enabling Controlled Spatial Sharing of Graphics and Compute on NVIDIA GPUs](https://mlsanity.com/paper/2605.01352/) (2026)
- [Energy-Aware Scheduling for Serverless LLM Serving on Shared GPUs](https://mlsanity.com/paper/2606.30391/) (2026)

## Abstract (arXiv)

GPU technology has been improving at an expedited pace in terms of size and performance, empowering HPC and AI/ML researchers to advance the scientific discovery process. However, this also leads to inefficient resource usage, as most GPU workloads, including complicated AI/ML models, are not able to utilize the GPU resources to their fullest extent -- encouraging support for GPU multi-tenancy. We propose MISO, a technique to exploit the Multi-Instance GPU (MIG) capability on the latest NVIDIA datacenter GPUs (e.g., A100, H100) to dynamically partition GPU resources among co-located jobs. MISO's key insight is to use the lightweight, more flexible Multi-Process Service (MPS) capability to predict the best MIG partition allocation for different jobs, without incurring the overhead of implementing them during exploration. Due to its ability to utilize GPU resources more efficiently, MISO achieves 49% and 16% lower average job completion time than the unpartitioned and optimal static GPU partition schemes, respectively.