mlsanityData to 14 Oct 2026

TensorFlow-Serving: Flexible, High-Performance ML Serving

arXiv 1712.06139 ↗First posted 17 Dec 2017cs.DC cs.LGcited by 85

Not on the map. Why?

Abstract (arXiv)

We describe TensorFlow-Serving, a system to serve machine learning models inside Google which is also available in the cloud and via open-source. It is extremely flexible in terms of the types of ML platforms it supports, and ways to integrate with systems that convey new models and updated versions from training to serving. At the same time, the core code paths around model lookup and inference have been carefully optimized to avoid performance pitfalls observed in naive implementations. Google uses it in many production deployments, including a multi-tenant model hosting service called TFS^2.

Key facts. Cited by 85 papers in the map (data through 14 Oct 2026). First posted on arXiv 17 Dec 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 11; 2022: 9; 2023: 22; 2024: 24; 2025: 13; 2026: 6.

85 papers.

Sort:

By area:

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
  3. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 152
  4. 2023Fast Distributed Inference Serving for Large Language Modelscited by 145
  5. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 99
  6. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  7. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Predictioncited by 66
  8. 2024Fast On-device LLM Inference with NPUscited by 62
  9. 2023Stateful Large Language Model Serving with Pensievecited by 61
  10. 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 55
  11. 2024MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Servingcited by 52
  12. 2024MegaAgent: A Large-Scale Autonomous LLM-based Multi-Agent System Without Predefined SOPscited by 47
38 more
  1. 2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Servicescited by 45
  2. 2024Queue management for slo-oriented large language model servingcited by 36
  3. 2022GEMEL: Model Merging for Memory-Efficient, Real-Time Video Analytics at the Edgecited by 32
  4. 2023AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Servingcited by 31
  5. 2021Fighting Gradients with Gradients: Dynamic Defenses against Adversarial Attackscited by 20
  6. 2021KAISA: An Adaptive Second-Order Optimizer Framework for Deep Neural Networkscited by 20
  7. 2023The Synergy of Speculative Decoding and Batching in Serving Large Language Modelscited by 19
  8. 2025Scaling Inference-Efficient Language Modelscited by 16
  9. 2023Apparate: Rethinking Early Exits to Tame Latency-Throughput Tensions in ML Servingcited by 15
  10. 2023The Pipeline for the Continuous Development of Artificial Intelligence Models -- Current State of Research and Practicecited by 14
  11. 2025HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Cloudscited by 14
  12. 2024Pie: Pooling CPU Memory for LLM Inferencecited by 13
  13. 2022Serving and Optimizing Machine Learning Workflows on Heterogeneous Infrastructurescited by 12
  14. 2023SuperServe: Fine-Grained Inference Serving for Unpredictable Workloadscited by 12
  15. 2024Towards Robust and Efficient Cloud-Edge Elastic Model Adaptation via Selective Entropy Distillationcited by 11
  16. 2021JIZHI: A Fast and Cost-Effective Model-As-A-Service System for Web-Scale Online Inference at Baiducited by 10
  17. 2021ModelCI-e: Enabling Continual Learning in Deep Learning Serving Systemscited by 9
  18. 2025LithOS: An Operating System for Efficient Machine Learning on GPUscited by 9
  19. 2025ELIS: Efficient LLM Iterative Scheduling System with Response Length Predictorcited by 9
  20. 2021Accelerating Deep Learning Inference via Learned Cachescited by 8
  21. 2021Serving DNN Models with Multi-Instance GPUs: A Case of the Reconfigurable Machine Scheduling Problemcited by 7
  22. 2021DeepRT: A Soft Real Time Scheduler for Computer Vision Applications on the Edgecited by 6
  23. 2022Deep Learning Workload Scheduling in GPU Datacenters: Taxonomy, Challenges and Visioncited by 6
  24. 2024SCAR: Scheduling Multi-Model AI Workloads on Heterogeneous Multi-Chiplet Module Acceleratorscited by 6
  25. 2021ApproxIFER: A Model-Agnostic Approach to Resilient and Robust Prediction Serving Systemscited by 5
  26. 2023Reconciling High Accuracy, Cost-Efficiency, and Low Latency of Inference Serving Systemscited by 5
  27. 2023Modyn: Data-Centric Machine Learning Pipeline Orchestrationcited by 5
  28. 2021Multi-model Machine Learning Inference Serving with GPU Spatial Partitioningcited by 4
  29. 2024Achieving Peak Performance for Large Language Models: A Systematic Reviewcited by 4
  30. 2023R'esum'e Parsing as Hierarchical Sequence Labeling: An Empirical Studycited by 3
  31. 2022PARIS and ELSA: An Elastic Scheduling Algorithm for Reconfigurable Multi-GPU Inference Serverscited by 2
  32. 2023BCEdge: SLO-Aware DNN Inference Services with Adaptive Batching on Edge Platformscited by 2
  33. 2023Symphony: Optimized DNN Model Serving using Deferred Batch Schedulingcited by 2
  34. 2023Deploying Deep Reinforcement Learning Systems: A Taxonomy of Challengescited by 2
  35. 2023Packrat: Automatic Reconfiguration for Latency Minimization in CPU-based DNN Servingcited by 2
  36. 2024Computing in the Era of Large Generative Models: From Cloud-Native to AI-Nativecited by 2
  37. 2024Scalable Artificial Intelligence for Science: Perspectives, Methods and Exemplarscited by 2
  38. 2024On the Cost of Model-Serving Frameworks: An Experimental Evaluationcited by 2

50 most cited of 85 All 85 (JSON)