# Exploring the Dynamic Scheduling Space of Real-Time Generative AI Applications on Emerging Heterogeneous Systems

arXiv: https://arxiv.org/abs/2507.14715 · HTML: https://mlsanity.com/paper/2507.14715/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 39 references that are in the map.
- First posted on arXiv 19 Jul 2025.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [Image Super-Resolution Using Very Deep Residual Channel Attention Networks](https://mlsanity.com/paper/1807.02758/) (2018)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU](https://mlsanity.com/paper/2303.06865/) (2023)
- [PointPainting: Sequential Fusion for 3D Object Detection](https://mlsanity.com/paper/1911.10150/) (2019)
- [Diffusion Models Are Real-Time Game Engines](https://mlsanity.com/paper/2408.14837/) (2024)
- [InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management](https://mlsanity.com/paper/2406.19707/) (2024)
- [Large Language Models and Games: A Survey and Roadmap](https://mlsanity.com/paper/2402.18659/) (2024)
- [Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models](https://mlsanity.com/paper/2402.07033/) (2024)
- [NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencing](https://mlsanity.com/paper/2403.00579/) (2024)
- [XRBench: An Extended Reality (XR) Machine Learning Benchmark Suite for the Metaverse](https://mlsanity.com/paper/2211.08675/) (2022)
- [Map-and-Conquer: Energy-Efficient Mapping of Dynamic Neural Nets onto Heterogeneous MPSoCs](https://mlsanity.com/paper/2302.12926/) (2023)
- [DREAM: A Dynamic Scheduler for Dynamic Real-time Multi-model ML Workloads](https://mlsanity.com/paper/2212.03414/) (2022)
- [SoftmAP: Software-Hardware Co-design for Integer-Only Softmax on Associative Processors](https://mlsanity.com/paper/2411.17847/) (2024)
- [Efficient Heterogeneous Video Segmentation at the Edge](https://mlsanity.com/paper/2208.11666/) (2022)

## Built on by (most cited first, 1 of 1)

- [Harmonizing Real-Time Constraints and Long-Horizon Reasoning: An Asynchronous Agentic Framework for Dynamic Scheduling](https://mlsanity.com/paper/2605.29262/) (2026)

## Abstract (arXiv)

The integration of generative AI models, particularly large language models (LLMs), into real-time multi-model AI applications such as video conferencing and gaming is giving rise to a new class of workloads: real-time generative AI (RTGen). These workloads combine the compute intensity and dynamic execution patterns of generative models with the stringent latency and concurrency constraints of real-time inference. To meet the diverse demands of RTGen workloads, modern edge platforms increasingly adopt heterogeneous system-on-chip (SoC) architectures that integrate CPUs, GPUs, and NPUs. Despite the potential of heterogeneous SoC, the scheduling space complexity and performance implications of RTGen workloads on such platforms remain underexplored. In this work, we perform a comprehensive characterization of RTGen workloads on AMD's latest heterogeneous SoC, Ryzen AI. We construct realistic multi-model scenarios inspired by industry use cases and profile model performance across all available backends. Using this data, we evaluate five scheduling policies and their impact on both real-time metrics (e.g., deadline violation rate) and LLM performance (e.g., time-to-first-token and tokens-per-second). Our results show that scheduling decisions significantly affect workload performance (e.g., leading to a 41.7% difference in deadline violation rates on average), and highlight the need for scheduling strategies that are aware of workload dynamics and hardware heterogeneity. Our findings underscore the importance of workload-aware, dynamic heterogeneous scheduling in enabling high-performance, on-device RTGen applications.