mlsanityData to 7 Oct 2026

Clipper: A Low-Latency Online Prediction Serving System

arXiv 1612.03079 ↗First posted 9 Dec 2016cs.DC cs.LGcited by 191

Not on the map. Why?

Cited by 30 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Machine learning is being deployed in a growing number of applications which demand real-time, accurate, and robust predictions under heavy query load. However, most machine learning frameworks and systems only address model training and not deployment. In this paper, we introduce Clipper, a general-purpose low-latency prediction serving system. Interposing between end-user applications and a wide range of machine learning frameworks, Clipper introduces a modular architecture to simplify model deployment across frameworks and applications. Furthermore, by introducing caching, batching, and adaptive model selection techniques, Clipper reduces prediction latency and improves prediction throughput, accuracy, and robustness without modifying the underlying machine learning frameworks. We evaluate Clipper on four common machine learning benchmark datasets and demonstrate its ability to meet the latency, accuracy, and throughput demands of online serving applications. Finally, we compare Clipper to the TensorFlow Serving system and demonstrate that we are able to achieve comparable throughput and latency while enabling model composition and online learning to improve accuracy and render more robust predictions.

Key facts. Cited by 191 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Dec 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 24; 2022: 27; 2023: 38; 2024: 39; 2025: 31; 2026: 32.

191 papers.

Sort:

By area:

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  3. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  4. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
  5. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 167
  6. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 146
  7. 2023Fast Distributed Inference Serving for Large Language Modelscited by 143
  8. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 96
  9. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instancescited by 83
  10. 2022Pathways: Asynchronous Distributed Dataflow for MLcited by 81
  11. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  12. 2023Punica: Multi-Tenant LoRA Servingcited by 80
38 more
  1. 2021Software Engineering for AI-Based Systems: A Surveycited by 68
  2. 2023Stateful Large Language Model Serving with Pensievecited by 59
  3. 2024MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Servingcited by 49
  4. 2024Queue management for slo-oriented large language model servingcited by 36
  5. 2025Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generationcited by 36
  6. 2022GEMEL: Model Merging for Memory-Efficient, Real-Time Video Analytics at the Edgecited by 32
  7. 2024Teola: Towards End-to-End Optimization of LLM-based Applicationscited by 32
  8. 2022Aryl: An Elastic Cluster Scheduler for Deep Learningcited by 27
  9. 2022Operationalizing Machine Learning: An Interview Studycited by 25
  10. 2022FedAdapter: Efficient Federated Learning for Modern NLPcited by 24
  11. 2021KAISA: An Adaptive Second-Order Optimizer Framework for Deep Neural Networkscited by 20
  12. 2022Merlin HugeCTR: GPU-accelerated Recommender System Training and Inferencecited by 20
  13. 2024SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Servingcited by 18
  14. 2023DeltaZip: Efficient Serving of Multiple Full-Model-Tuned LLMscited by 17
  15. 2022End-to-end Optimization of Machine Learning Prediction Queriescited by 16
  16. 2022Cloud-Device Collaborative Adaptation to Continual Changing Environments in the Real-worldcited by 16
  17. 2025xRouter: Training Cost-Aware LLMs Orchestration System via Reinforcement Learningcited by 16
  18. 2023ERASER: Machine Unlearning in MLaaS via an Inference Serving-Aware Approachcited by 15
  19. 2023Apparate: Rethinking Early Exits to Tame Latency-Throughput Tensions in ML Servingcited by 15
  20. 2022Management of Machine Learning Lifecycle Artifacts: A Surveycited by 13
  21. 2024Ads Recommendation in a Collapsed and Entangled Worldcited by 13
  22. 2024Toward Sustainable GenAI using Generation Directives for Carbon-Friendly Large Language Model Inferencecited by 13
  23. 2022Serving and Optimizing Machine Learning Workflows on Heterogeneous Infrastructurescited by 12
  24. 2023SuperServe: Fine-Grained Inference Serving for Unpredictable Workloadscited by 12
  25. 2022Multi-user Co-inference with Batch Processing Capable Edge Servercited by 11
  26. 2023Quiver: Supporting GPUs for Low-Latency, High-Throughput GNN Serving with Workload Awarenesscited by 11
  27. 2023Cloud-Device Collaborative Learning for Multimodal Large Language Modelscited by 11
  28. 2024Towards Robust and Efficient Cloud-Edge Elastic Model Adaptation via Selective Entropy Distillationcited by 11
  29. 2024Aladdin: Joint Placement and Scaling for SLO-Aware LLM Servingcited by 11
  30. 2021A Serverless Cloud-Fog Platform for DNN-Based Video Analytics with Incremental Learningcited by 9
  31. 2021ModelCI-e: Enabling Continual Learning in Deep Learning Serving Systemscited by 9
  32. 2022A GPU-specialized Inference Parameter Server for Large-Scale Deep Recommendation Modelscited by 9
  33. 2024FlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guaranteescited by 9
  34. 2024Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Surveycited by 9
  35. 2024Etalon: Holistic Performance Evaluation Framework for LLM Inference Systemscited by 9
  36. 2025LithOS: An Operating System for Efficient Machine Learning on GPUscited by 9
  37. 2021Accelerating Deep Learning Inference via Learned Cachescited by 8
  38. 2021When in Doubt, Summon the Titans: Efficient Inference with Large Modelscited by 8

50 most cited of 191 All 191 (JSON)