mlsanityData to 7 Oct 2026

Serving DNNs like Clockwork: Performance Predictability from the Bottom Up

arXiv 2006.02464 ↗First posted 3 Jun 2020cs.DC cs.LGcited by 119

Not on the map. Why?

Cited by 32 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Machine learning inference is becoming a core building block for interactive web applications. As a result, the underlying model serving systems on which these applications depend must consistently meet low latency targets. Existing model serving architectures use well-known reactive techniques to alleviate common-case sources of latency, but cannot effectively curtail tail latency caused by unpredictable execution times. Yet the underlying execution times are not fundamentally unpredictable - on the contrary we observe that inference using Deep Neural Network (DNN) models has deterministic performance. Here, starting with the predictable execution times of individual DNN inferences, we adopt a principled design methodology to successively build a fully distributed model serving system that achieves predictable end-to-end performance. We evaluate our implementation, Clockwork, using production trace workloads, and show that Clockwork can support thousands of models while simultaneously meeting 100ms latency targets for 99.9999% of requests. We further demonstrate that Clockwork exploits predictable execution times to achieve tight request-level service-level objectives (SLOs) as well as a high degree of request-level performance isolation.

Key facts. Cited by 119 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 6; 2023: 26; 2024: 31; 2025: 23; 2026: 29.

119 papers.

Sort:

By area:

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  3. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  4. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  5. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  6. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 167
  7. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 146
  8. 2023Fast Distributed Inference Serving for Large Language Modelscited by 143
  9. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 96
  10. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instancescited by 83
  11. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  12. 2023Punica: Multi-Tenant LoRA Servingcited by 80
38 more
  1. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 78
  2. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Predictioncited by 64
  3. 2025StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generationcited by 56
  4. 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 54
  5. 2024MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Servingcited by 49
  6. 2024MegaAgent: A Large-Scale Autonomous LLM-based Multi-Agent System Without Predefined SOPscited by 45
  7. 2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Servicescited by 44
  8. 2024Optimizing RLHF Training for Large Language Models with Stage Fusioncited by 41
  9. 2024Queue management for slo-oriented large language model servingcited by 36
  10. 2025Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generationcited by 36
  11. 2022GEMEL: Model Merging for Memory-Efficient, Real-Time Video Analytics at the Edgecited by 32
  12. 2024Teola: Towards End-to-End Optimization of LLM-based Applicationscited by 32
  13. 2022Aryl: An Elastic Cluster Scheduler for Deep Learningcited by 27
  14. 2023Reducing Energy Bloat in Large Model Trainingcited by 22
  15. 2023DeltaZip: Efficient Serving of Multiple Full-Model-Tuned LLMscited by 17
  16. 2023ERASER: Machine Unlearning in MLaaS via an Inference Serving-Aware Approachcited by 15
  17. 2023Apparate: Rethinking Early Exits to Tame Latency-Throughput Tensions in ML Servingcited by 15
  18. 2025Scaling Inference-Efficient Language Modelscited by 14
  19. 2022SplitPlace: AI Augmented Splitting and Placement of Large-Scale Neural Networks in Mobile Edge Environmentscited by 11
  20. 2023Quiver: Supporting GPUs for Low-Latency, High-Throughput GNN Serving with Workload Awarenesscited by 11
  21. 2024Training and Serving System of Foundation Models: A Comprehensive Surveycited by 11
  22. 2024Aladdin: Joint Placement and Scaling for SLO-Aware LLM Servingcited by 11
  23. 2023RT-LM: Uncertainty-Aware Resource Management for Real-Time Inference of Language Modelscited by 9
  24. 2024FlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guaranteescited by 9
  25. 2024Etalon: Holistic Performance Evaluation Framework for LLM Inference Systemscited by 9
  26. 2025LithOS: An Operating System for Efficient Machine Learning on GPUscited by 9
  27. 2022DREAM: A Dynamic Scheduler for Dynamic Real-time Multi-model ML Workloadscited by 8
  28. 2025HELIOS: Adaptive Model And Early-Exit Selection for Efficient LLM Inference Servingcited by 8
  29. 2025ELIS: Efficient LLM Iterative Scheduling System with Response Length Predictorcited by 8
  30. 2021Serving DNN Models with Multi-Instance GPUs: A Case of the Reconfigurable Machine Scheduling Problemcited by 7
  31. 2022Deep Learning Workload Scheduling in GPU Datacenters: Taxonomy, Challenges and Visioncited by 6
  32. 2025Niyama : Breaking the Silos of LLM Inference Servingcited by 6
  33. 2025FairBatching: Fairness-Aware Batch Formation for LLM Inferencecited by 6
  34. 2023Reconciling High Accuracy, Cost-Efficiency, and Low Latency of Inference Serving Systemscited by 5
  35. 2023MOSEL: Inference Serving Using Dynamic Modality Selectioncited by 5
  36. 2023PolyThrottle: Energy-efficient Neural Network Inference on Edge Devicescited by 5
  37. 2024SwiftDiffusion: Efficient Diffusion Model Serving with Add-on Modulescited by 5
  38. 2024ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Servingcited by 5

50 most cited of 119 All 119 (JSON)