# Strait: Perceiving Priority and Interference in ML Inference Serving

arXiv: https://arxiv.org/abs/2604.28175 · HTML: https://mlsanity.com/paper/2604.28175/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 94 references that are in the map.
- First posted on arXiv 30 Apr 2026.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [On the Convergence of Adam and Beyond](https://mlsanity.com/paper/1904.09237/) (2019)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [MLPerf Inference Benchmark](https://mlsanity.com/paper/1911.02549/) (2019)
- [Clipper: A Low-Latency Online Prediction Serving System](https://mlsanity.com/paper/1612.03079/) (2016)
- [Serving DNNs like Clockwork: Performance Predictability from the Bottom Up](https://mlsanity.com/paper/2006.02464/) (2020)
- [TensorFlow-Serving: Flexible, High-Performance ML Serving](https://mlsanity.com/paper/1712.06139/) (2017)
- [Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider](https://mlsanity.com/paper/2003.03423/) (2020)
- [Forecasting GPU Performance for Deep Learning Training and Inference](https://mlsanity.com/paper/2407.13853/) (2024)
- [LithOS: An Operating System for Efficient Machine Learning on GPUs](https://mlsanity.com/paper/2504.15465/) (2025)
- [A Survey of Multi-Tenant Deep Learning Inference on GPU](https://mlsanity.com/paper/2203.09040/) (2022)
- [SGDRC: Software-Defined Dynamic Resource Control for Concurrent DNN Inference on NVIDIA GPUs](https://mlsanity.com/paper/2407.13996/) (2024)
- [PolyThrottle: Energy-efficient Neural Network Inference on Edge Devices](https://mlsanity.com/paper/2310.19991/) (2023)
- [ADEPOS: Anomaly Detection based Power Saving for Predictive Maintenance using Edge Computing](https://mlsanity.com/paper/1811.00873/) (2018)
- [A Unified Optimization Approach for CNN Model Inference on Integrated GPUs](https://mlsanity.com/paper/1907.02154/) (2019)
- [Characterizing Concurrency Mechanisms for NVIDIA GPUs under Deep Learning Workloads](https://mlsanity.com/paper/2110.00459/) (2021)
- [High-Throughput, High-Performance Deep Learning-Driven Light Guide Plate Surface Visual Quality Inspection Tailored for Real-World Manufacturing Environments](https://mlsanity.com/paper/2212.10632/) (2022)
- [A House United Within Itself: SLO-Awareness for On-Premises Containerized ML Inference Clusters via Faro](https://mlsanity.com/paper/2409.19488/) (2024)
- [ML Inference Scheduling with Predictable Latency](https://mlsanity.com/paper/2512.18725/) (2025)

## Abstract (arXiv)

Machine learning (ML) inference serving systems host deep neural network (DNN) models and schedule incoming inference requests across deployed GPUs. However, limited support for task prioritization and insufficient latency estimation under concurrent execution may restrict their applicability in on-premises scenarios. We present Strait, a serving system designed to enhance deadline satisfaction for dual-priority inference traffic under high GPU utilization. To improve latency estimation, Strait models potential contention during data transfer and accounts for kernel execution interference through an adaptive prediction model. By drawing on these predictions, it performs priority-aware scheduling to deliver differentiated handling. Evaluation results under intense workloads suggest that Strait reduces deadline violations for high-priority tasks by 1.02 to 11.18 percentage points while incurring acceptable costs on low-priority tasks. Compared to software-defined preemption approaches, Strait also exhibits more equitable performance.