# DREAM: A Dynamic Scheduler for Dynamic Real-time Multi-model ML Workloads

arXiv: https://arxiv.org/abs/2212.03414 · HTML: https://mlsanity.com/paper/2212.03414/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 51 references that are in the map.
- First posted on arXiv 7 Dec 2022.

## Builds on (most cited first, 21 of 21)

- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search](https://mlsanity.com/paper/1812.03443/) (2018)
- [SkipNet: Learning Dynamic Routing in Convolutional Networks](https://mlsanity.com/paper/1711.09485/) (2017)
- [MLPerf Inference Benchmark](https://mlsanity.com/paper/1911.02549/) (2019)
- [BlockDrop: Dynamic Inference Paths in Residual Networks](https://mlsanity.com/paper/1711.08393/) (2017)
- [Convolutional Networks with Adaptive Inference Graphs](https://mlsanity.com/paper/1711.11503/) (2017)
- [SOSNet: Second Order Similarity Regularization for Local Descriptor Learning](https://mlsanity.com/paper/1904.05019/) (2019)
- [A Large-Scale Car Dataset for Fine-Grained Categorization and Verification](https://mlsanity.com/paper/1506.08959/) (2015)
- [Serving DNNs like Clockwork: Performance Predictability from the Bottom Up](https://mlsanity.com/paper/2006.02464/) (2020)
- [Deep Residual Learning for Small-Footprint Keyword Spotting](https://mlsanity.com/paper/1710.10361/) (2017)
- [PREMA: A Predictive Multi-task Scheduling Algorithm For Preemptible Neural Processing Units](https://mlsanity.com/paper/1909.04548/) (2019)
- [Learning Depth from Single Images with Deep Neural Network Embedding Focal Length](https://mlsanity.com/paper/1803.10039/) (2018)
- [MAGMA: An Optimization Framework for Mapping Multiple DNNs on Multiple Accelerator Cores](https://mlsanity.com/paper/2104.13997/) (2021)
- [VELTAIR: Towards High-Performance Multi-tenant Deep Learning Services via Adaptive Compilation and Scheduling](https://mlsanity.com/paper/2201.06212/) (2022)
- [XRBench: An Extended Reality (XR) Machine Learning Benchmark Suite for the Metaverse](https://mlsanity.com/paper/2211.08675/) (2022)
- [EyeCoD: Eye Tracking System Acceleration via FlatCam-based Algorithm & Accelerator Co-Design](https://mlsanity.com/paper/2206.00877/) (2022)
- [RAPID-RL: A Reconfigurable Architecture with Preemptive-Exits for Efficient Deep-Reinforcement Learning](https://mlsanity.com/paper/2109.08231/) (2021)

## Built on by (most cited first, 8 of 8)

- [SCAR: Scheduling Multi-Model AI Workloads on Heterogeneous Multi-Chiplet Module Accelerators](https://mlsanity.com/paper/2405.00790/) (2024)
- [MTP: A Meaning-Typed Language Abstraction for AI-Integrated Programming](https://mlsanity.com/paper/2405.08965/) (2024)
- [Performance Implications of Multi-Chiplet Neural Processing Units on Autonomous Driving Perception](https://mlsanity.com/paper/2411.16007/) (2024)
- [Exploring the Dynamic Scheduling Space of Real-Time Generative AI Applications on Emerging Heterogeneous Systems](https://mlsanity.com/paper/2507.14715/) (2025)
- [CaMDN: Enhancing Cache Efficiency for Multi-tenant DNNs on Integrated NPUs](https://mlsanity.com/paper/2505.06625/) (2025)
- [Orchestrating Multimodal DNN Workloads in Wireless Neural Processing](https://mlsanity.com/paper/2603.02109/) (2026)
- [AESOP: Adversarial Execution-path Selection to Overload Deep Learning Pipelines](https://mlsanity.com/paper/2605.10987/) (2026)
- [Terastal: Layer-Variant-based Scheduling for Real-Time Multi-DNN Workloads on Heterogeneous Accelerators](https://mlsanity.com/paper/2606.06818/) (2026)

## Abstract (arXiv)

Emerging real-time multi-model ML (RTMM) workloads such as AR/VR and drone control involve dynamic behaviors in various granularity; task, model, and layers within a model. Such dynamic behaviors introduce new challenges to the system software in an ML system since the overall system load is not completely predictable, unlike traditional ML workloads. In addition, RTMM workloads require real-time processing, involve highly heterogeneous models, and target resource-constrained devices. Under such circumstances, developing an effective scheduler gains more importance to better utilize underlying hardware considering the unique characteristics of RTMM workloads. Therefore, we propose a new scheduler, DREAM, which effectively handles various dynamicity in RTMM workloads targeting multi-accelerator systems. DREAM quantifies the unique requirements for RTMM workloads and utilizes the quantified scores to drive scheduling decisions, considering the current system load and other inference jobs on different models and input frames. DREAM utilizes tunable parameters that provide fast and effective adaptivity to dynamic workload changes. In our evaluation of five scenarios of RTMM workload, DREAM reduces the overall UXCost, which is an equivalent metric of the energy-delay product (EDP) for RTMM defined in the paper, by 32.2% and 50.0% in the geometric mean (up to 80.8% and 97.6%) compared to state-of-the-art baselines, which shows the efficacy of our scheduling methodology.