mlsanityData to 7 Oct 2026

PREMA: A Predictive Multi-task Scheduling Algorithm For Preemptible Neural Processing Units

arXiv 1909.04548 ↗First posted 6 Sep 2019cs.DC cs.LG cs.NEcited by 30

Not on the map. Why?

Abstract (arXiv)

To amortize cost, cloud vendors providing DNN acceleration as a service to end-users employ consolidation and virtualization to share the underlying resources among multiple DNN service requests. This paper makes a case for a "preemptible" neural processing unit (NPU) and a "predictive" multi-task scheduler to meet the latency demands of high-priority inference while maintaining high throughput. We evaluate both the mechanisms that enable NPUs to be preemptible and the policies that utilize them to meet scheduling objectives. We show that preemptive NPU multi-tasking can achieve an average 7.8x, 1.4x, and 4.8x improvement in latency, throughput, and SLA satisfaction, respectively.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 7; 2023: 5; 2024: 6; 2025: 2; 2026: 2.

30 papers.

Sort:

By area:

  1. 2021Auto-Split: A General Framework of Collaborative Edge-Cloud AIcited by 29
  2. 2021Energon: Towards Efficient Acceleration of Transformers Using Dynamic Sparse Attentioncited by 26
  3. 2023vTrain: A Simulation Framework for Evaluating Cost-effective and Compute-optimal Large Language Model Trainingcited by 22
  4. 2021MAGMA: An Optimization Framework for Mapping Multiple DNNs on Multiple Accelerator Corescited by 15
  5. 2025Agent.xpu: Efficient Scheduling of Agentic LLM Workloads on Heterogeneous SoCcited by 11
  6. 2024FlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guaranteescited by 9
  7. 2021RecPipe: Co-designing Models and Hardware to Jointly Optimize Recommendation Quality and Performancecited by 8
  8. 2022DREAM: A Dynamic Scheduler for Dynamic Real-time Multi-model ML Workloadscited by 8
  9. 2023MoCA: Memory-Centric, Adaptive Execution for Multi-Tenant Deep Neural Networkscited by 8
  10. 2023Sparse-DySta: Sparsity-Aware Dynamic and Static Scheduling for Sparse Multi-DNN Workloadscited by 6
  11. 2021A Survey of Large-Scale Deep Learning Serving System Optimization: Challenges and Opportunitiescited by 5
  12. 2021Multi-model Machine Learning Inference Serving with GPU Spatial Partitioningcited by 4
18 more
  1. 2022Scale-out Systolic Arrayscited by 4
  2. 2024Deep Reinforcement Learning based Online Scheduling Policy for Deep Neural Network Multi-Tenant Multi-Accelerator Systemscited by 4
  3. 2022Multi-DNN Accelerators for Next-Generation AI Systemscited by 3
  4. 2022DiVa: An Accelerator for Differentially Private Machine Learningcited by 3
  5. 2021How to Reach Real-Time AI on Consumer Devices? Solutions for Programmable and Custom Architecturescited by 2
  6. 2022PARIS and ELSA: An Elastic Scheduling Algorithm for Reconfigurable Multi-GPU Inference Serverscited by 2
  7. 2023BCEdge: SLO-Aware DNN Inference Services with Adaptive Batching on Edge Platformscited by 2
  8. 2024Hardware-Assisted Virtualization of Neural Processing Units for Cloud Platformscited by 2
  9. 2021GPU Domain Specialization via Composable On-Package Architecturecited by 1
  10. 2022EcoFlow: Efficient Convolutional Dataflows for Low-Power Neural Network Acceleratorscited by 1
  11. 2024Obsidian: Cooperative State-Space Exploration for Performant Inference on Secure ML Acceleratorscited by 1
  12. 2022A Mixed Quantization Network for Computationally Efficient Mobile Inverse Tone Mappingcited by 0
  13. 2023Hera: A Heterogeneity-Aware Multi-Tenant Inference Server for Personalized Recommendationscited by 0
  14. 2024Towards Fair and Firm Real-Time Scheduling in DNN Multi-Tenant Multi-Accelerator Systems via Reinforcement Learningcited by 0
  15. 2024PREBA: A Hardware/Software Co-Design for Multi-Instance GPU based AI Inference Serverscited by 0
  16. 2025CaMDN: Enhancing Cache Efficiency for Multi-tenant DNNs on Integrated NPUscited by 0
  17. 2026Terastal: Layer-Variant-based Scheduling for Real-Time Multi-DNN Workloads on Heterogeneous Acceleratorscited by 0
  18. 2026DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMscited by 0