mlsanityData to 7 Oct 2026

Fast Distributed Inference Serving for Large Language Models

arXiv 2305.05920 ↗First posted 10 May 2023cs.LG cs.DCcited by 143

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large language models (LLMs) power a new generation of interactive AI applications exemplified by ChatGPT. The interactive nature of these applications demands low latency for LLM inference. Existing LLM serving systems use run-to-completion processing for inference jobs, which suffers from head-of-line blocking and long latency. We present FastServe, a distributed inference serving system for LLMs. FastServe exploits the autoregressive pattern of LLM inference to enable preemption at the granularity of each output token. FastServe uses preemptive scheduling to minimize latency with a novel skip-join Multi-Level Feedback Queue scheduler. Based on the new semi-information-agnostic setting of LLM inference, the scheduler leverages the input length information to assign an appropriate initial queue for each arrival job to join. The higher priority queues than the joined queue are skipped to reduce demotions. We design an efficient GPU memory management mechanism that proactively offloads and uploads intermediate state between GPU memory and host memory for LLM inference. We build a system prototype of FastServe and experimental results show that compared to the state-of-the-art solution vLLM, FastServe improves the throughput by up to 31.4x and 17.9x under the same average and tail latency requirements, respectively.

Key facts. Cited by 143 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 77 references that are in the map. First posted on arXiv 10 May 2023.

Builds on

25 of 77 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  10. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  11. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  12. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
13 more
  1. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  2. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  3. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  4. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
  5. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  6. 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
  7. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 119
  8. 2017TensorFlow-Serving: Flexible, High-Performance ML Servingcited by 85
  9. 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 84
  10. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71
  11. 2022MPCFormer: fast, performant and private Transformer inference with MPCcited by 67
  12. 2022Fine-tuning Language Models over Slow Networks using Activation Compression with Guaranteescited by 29
  13. 2013BigDataBench: a Big Data Benchmark Suite from Web Search Enginescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 46; 2025: 49; 2026: 42.

143 papers.

Sort:

By area:

  1. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  2. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  3. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  4. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  5. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  6. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
  7. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 167
  8. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  9. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  10. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  11. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 97
  12. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 93
38 more
  1. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  2. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systemscited by 84
  3. 2024Efficient LLM Scheduling by Learning to Rankcited by 83
  4. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  5. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 78
  6. 2023Fairness in Serving Large Language Modelscited by 77
  7. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 68
  8. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 67
  9. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Predictioncited by 64
  10. 2025Autellix: An Efficient Serving Engine for LLM Agents as General Programscited by 57
  11. 2024Efficiently Scaling LLM Reasoning with Certaindexcited by 56
  12. 2025KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflowscited by 56
  13. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 46
  14. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Livecited by 43
  15. 2024Optimizing RLHF Training for Large Language Models with Stage Fusioncited by 41
  16. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inferencecited by 37
  17. 2024Don't Stop Me Now: Embedding Based Scheduling for LLMscited by 34
  18. 2024InferCept: Efficient Intercept Support for Augmented Large Language Model Inferencecited by 32
  19. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  20. 2024M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneitycited by 30
  21. 2024LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scalecited by 30
  22. 2025ServeGen: Workload Characterization and Generation of Large Language Model Serving in Productioncited by 29
  23. 2023Compress, Then Prompt: Improving Accuracy-Efficiency Trade-off of LLM Inference with Transferable Promptcited by 28
  24. 2023Integration of Large Language Models and Federated Learningcited by 28
  25. 2025TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platformscited by 28
  26. 2024P/D-Serve: Serving Disaggregated Large Language Model at Scalecited by 27
  27. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 21
  28. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUscited by 21
  29. 2024BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batchingcited by 19
  30. 2024SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Servingcited by 18
  31. 2025Online Scheduling for LLM Inference with KV Cache Constraintscited by 17
  32. 2025JITServe: SLO-aware LLM Serving with Imprecise Request Informationcited by 17
  33. 2025semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storagecited by 15
  34. 2026TraceLab: Characterizing Coding Agent Workloads for LLM Servingcited by 14
  35. 2025IC-Cache: Efficient Large Language Model Serving via In-context Cachingcited by 13
  36. 2025SLED: A Speculative LLM Decoding Framework for Efficient Edge Servingcited by 13
  37. 2024Efficient Heterogeneous Large Language Model Decoding with Model-Attention Disaggregationcited by 12
  38. 2025HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Cloudscited by 12

50 most cited of 143 All 143 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs
  2. 2024AQUA: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains
  3. 2025Taming the Titans: A Survey of Efficient LLM Inference Serving
  4. 2025Characterizing Communication Patterns in Distributed Large Language Model Inference
  5. 2025Throughput-Optimal Scheduling Algorithms for LLM Inference and AI Agents
  6. 2024Inference Performance Optimization for Large Language Models on CPUs
  7. 2024GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments
  8. 2024ALISE: Accelerating Large Language Model Serving with Speculative Scheduling