# MeanField Surrogate Modeling for Scalable Runtime Scheduling of Concurrent Heterogeneous AI Inference on Shared GPUs

arXiv: https://arxiv.org/abs/2609.02109 · HTML: https://mlsanity.com/paper/2609.02109/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 10 references that are in the map.
- First posted on arXiv 2 Sep 2026.

## Builds on (most cited first, 6 of 6)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Clipper: A Low-Latency Online Prediction Serving System](https://mlsanity.com/paper/1612.03079/) (2016)
- [Serving DNNs like Clockwork: Performance Predictability from the Bottom Up](https://mlsanity.com/paper/2006.02464/) (2020)
- [Automated Runtime-Aware Scheduling for Multi-Tenant DNN Inference on GPU](https://mlsanity.com/paper/2111.14255/) (2021)
- [DVFS-Aware DNN Inference on GPUs: Latency Modeling and Performance Analysis](https://mlsanity.com/paper/2502.06295/) (2025)
- [Miriam: Exploiting Elastic Kernels for Real-time Multi-DNN Inference on Edge GPU](https://mlsanity.com/paper/2307.04339/) (2023)

## Abstract (arXiv)

Deploying heterogeneous AI models concurrently on a shared GPU introduces resource contention that complicates runtime scheduling. While surrogate models avoid costly online benchmarking, their profiling requirements typically grow combinatorially with the number of co-running models, limiting scalability. We propose a MeanField surrogate that predicts per-model performance from local configuration and aggregate GPU state rather than explicitly modeling all joint interactions. Experiments on concurrent LLM and vision workloads across N ∈ 2,3,4,5,6 show high predictive accuracy (R2 ≈ 0.96) with an empirical sample budget that grows approximately linearly in N, in contrast to the combinatorial cost of fully joint profiling. Integrated into a genetic algorithm scheduler, the surrogate scales to an N=5 problem with 78,732 feasible joint configurations, remaining within 0.10% of the exhaustive search with zero SLA violations across eight dynamic workload scenarios, while complete online GA decisions take 26 ms median, about 5× faster than exhaustive surrogate search.