# Miriam: Exploiting Elastic Kernels for Real-time Multi-DNN Inference on Edge GPU

arXiv: https://arxiv.org/abs/2307.04339 · HTML: https://mlsanity.com/paper/2307.04339/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 52 references that are in the map.
- First posted on arXiv 10 Jul 2023.

## Builds on (most cited first, 12 of 12)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer](https://mlsanity.com/paper/2110.02178/) (2021)
- [Residual Attention Network for Image Classification](https://mlsanity.com/paper/1704.06904/) (2017)
- [Gate-Variants of Gated Recurrent Unit (GRU) Neural Networks](https://mlsanity.com/paper/1701.05923/) (2017)
- [Ansor: Generating High-Performance Tensor Programs for Deep Learning](https://mlsanity.com/paper/2006.06762/) (2020)
- [NestDNN: Resource-Aware Multi-Tenant On-Device Deep Learning for Continuous Mobile Vision](https://mlsanity.com/paper/1810.10090/) (2018)
- [IOS: Inter-Operator Scheduler for CNN Acceleration](https://mlsanity.com/paper/2011.01302/) (2020)
- [VELTAIR: Towards High-Performance Multi-tenant Deep Learning Services via Adaptive Compilation and Scheduling](https://mlsanity.com/paper/2201.06212/) (2022)
- [Dynamic Space-Time Scheduling for GPU Inference](https://mlsanity.com/paper/1901.00041/) (2018)
- [Automated Runtime-Aware Scheduling for Multi-Tenant DNN Inference on GPU](https://mlsanity.com/paper/2111.14255/) (2021)
- [Tango: A Deep Neural Network Benchmark Suite for Various Accelerators](https://mlsanity.com/paper/1901.04987/) (2019)

## Built on by (most cited first, 5 of 5)

- [Panopticus: Omnidirectional 3D Object Detection on Resource-constrained Edge Devices](https://mlsanity.com/paper/2410.01270/) (2024)
- [Embodied.cpp: A Portable Inference Runtime of Embodied AI Models on Heterogeneous Robots](https://mlsanity.com/paper/2607.02501/) (2026)
- [Responsive DNN Adaptation for Video Analytics against Environment Shift via Hierarchical Mobile-Cloud Collaborations](https://mlsanity.com/paper/2505.00745/) (2025)
- [Covariance-Guided Resource Adaptive Learning for Efficient Edge Inference](https://mlsanity.com/paper/2603.14577/) (2026)
- [MeanField Surrogate Modeling for Scalable Runtime Scheduling of Concurrent Heterogeneous AI Inference on Shared GPUs](https://mlsanity.com/paper/2609.02109/) (2026)

## Abstract (arXiv)

Many applications such as autonomous driving and augmented reality, require the concurrent running of multiple deep neural networks (DNN) that poses different levels of real-time performance requirements. However, coordinating multiple DNN tasks with varying levels of criticality on edge GPUs remains an area of limited study. Unlike server-level GPUs, edge GPUs are resource-limited and lack hardware-level resource management mechanisms for avoiding resource contention. Therefore, we propose Miriam, a contention-aware task coordination framework for multi-DNN inference on edge GPU. Miriam consolidates two main components, an elastic-kernel generator, and a runtime dynamic kernel coordinator, to support mixed critical DNN inference. To evaluate Miriam, we build a new DNN inference benchmark based on CUDA with diverse representative DNN workloads. Experiments on two edge GPU platforms show that Miriam can increase system throughput by 92% while only incurring less than 10% latency overhead for critical tasks, compared to state of art baselines.