mlsanityData to 7 Oct 2026

ServeGen: Workload Characterization and Generation of Large Language Model Serving in Production

arXiv 2505.09999 ↗First posted 15 May 2025cs.DCcited by 29

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

With the widespread adoption of Large Language Models (LLMs), serving LLM inference requests has become an increasingly important task, attracting active research advancements. Practical workloads play an essential role in this process: they are critical for motivating and benchmarking serving techniques and systems. However, the existing understanding of real-world LLM serving workloads is limited due to the lack of a comprehensive workload characterization. Prior analyses remain insufficient in scale and scope, thus failing to fully capture intricate workload characteristics. In this paper, we fill the gap with an in-depth characterization of LLM serving workloads collected from our worldwide cloud inference serving service, covering not only language models but also emerging multimodal and reasoning models, and unveiling important new findings in each case. Moreover, based on our findings, we propose ServeGen, a principled framework for generating realistic LLM serving workloads by composing them on a per-client basis. A practical use case in production validates that ServeGen avoids 50% under-provisioning compared to naive workload generation, demonstrating ServeGen's advantage in performance benchmarking. ServeGen is available at https://github.com/alibaba/ServeGen.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 29 of its 62 references that are in the map. First posted on arXiv 15 May 2025.

Builds on

29 of 62 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storage28 Apr
  2. 2025Locality-aware Fair Scheduling in LLM Serving24 Jan
  3. 2024P/D-Serve: Serving Disaggregated Large Language Model at Scale15 Aug
  4. 2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services25 Apr
  5. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism15 Apr
  6. 2024MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving2 Apr
  7. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  8. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems31 Jan
  9. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  10. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 19

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  5. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  6. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 787
  7. 2025Qwen2.5-Omni Technical Reportcited by 704
  8. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  9. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  10. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 115
  11. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  12. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 106
7 more
  1. 2019Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloadscited by 79
  2. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 78
  3. 2020Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Providercited by 65
  4. 2023ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scalecited by 47
  5. 2024Characterization of Large Language Model Development in the Datacentercited by 46
  6. 2021Characterization and Prediction of Deep Learning Workloads in Large-Scale GPU Datacenterscited by 31
  7. 2019Characterizing Deep Learning Training Workloads on Alibaba-PAIcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 25.

29 papers.

Sort:

By area:

  1. 2025Cornfigurator: Automated Planning for Any-to-Any Multimodal Model Servingcited by 4
  2. 2025WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Servingcited by 2
  3. 2025Role-Based Fault Tolerance System for LLM RL Post-Trainingcited by 2
  4. 2026Simple is Better: Multiplication May Be All You Need for LLM Request Schedulingcited by 2
  5. 2026CRAFT: Fine-Grained Cost-Aware Expert Replication For Efficient Mixture-of-Experts Servingcited by 2
  6. 2026Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inferencecited by 2
  7. 2026Nitsum: Serving Tiered LLM Requests with Adaptive Tensor Parallelismcited by 2
  8. 2026SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Schedulingcited by 2
  9. 2025MeAJOR Corpus: A Multi-Source Dataset for Phishing Email Detectioncited by 1
  10. 2026TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inferencecited by 1
  11. 2026Cornserve: A Distributed Serving System for Any-to-Any Multimodal Modelscited by 1
  12. 2026The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Projectcited by 1
17 more
  1. 2026Dual-Pool Token-Budget Routing for Cost-Efficient and Reliable LLM Servingcited by 1
  2. 2026ObjectCache: Layerwise Object-Storage Retrieval for KV Cache Reusecited by 1
  3. 2026Diagnose Before You Compress: Prediction-Independent Bottleneck Witness Refinement for LLM Serving Tracescited by 1
  4. 2026Hydra: Phase-Aware Workload Characterization of LLM Inference across Edge SoC Generations, Backends, and Quantization Levelscited by 1
  5. 2026Hardware-Software Co-design for 3D-DRAM-based LLM Serving Acceleratorcited by 0
  6. 2026Regulating Branch Parallelism in LLM Servingcited by 0
  7. 2026Geometry-Aware Online Scheduling for LLM Serving: From Theoretical Bound to System Practicecited by 0
  8. 2026ASAP: A Disaggregated and Asynchronous Inference System for MoE Prefillcited by 0
  9. 2026Energy-Aware Scheduling for Serverless LLM Serving on Shared GPUscited by 0
  10. 2026NELSSA: A GPU-PNM Heterogeneous System for Mixed-Length LLM Serving via Length-based Request Placementcited by 0
  11. 2026InFactPlanner: Planning Sustainable Geo-Distributed LLM Data Centerscited by 0
  12. 2026A Year in LLM Serving: Workload Evolution, Caching and Load-Balancingcited by 0
  13. 2026Pallas: A Proactive KV Cache Migration Framework for LLM Inference in AI-RANcited by 0
  14. 2026Token Latency Fairness: Performance Isolation for Multi-Tenant LLM Servingcited by 0
  15. 2026SARA: SLO-Aware Resource Allocation for Disaggregated Agentic LLM Servicescited by 0
  16. 2026PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decodingcited by 0
  17. 2026Inference-Layer Security: Defending Against Adversarial Inference and Infrastructure Abusecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Comparative Analysis of Large Language Model Inference Serving Systems: A Performance Study of vLLM and HuggingFace TGI
  2. 2024ScaleLLM: A Resource-Frugal LLM Serving Framework by Optimizing End-to-End Efficiency
  3. 2026LLM Serving in the Wild: An Empirical Study of Frameworks, Methods, and System Designs
  4. 2024GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments
  5. 2026FineServe: A Fine-Grained Dataset and Characterization of Global LLM Serving Workloads
  6. 2024The CAP Principle for LLM Serving: A Survey of Long-Context Large Language Model Serving
  7. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving
  8. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems