# ServeGen: Workload Characterization and Generation of Large Language Model Serving in Production

arXiv: https://arxiv.org/abs/2505.09999 · HTML: https://mlsanity.com/paper/2505.09999/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving.
- Builds on 29 of its 62 references that are in the map.
- First posted on arXiv 15 May 2025.

## Placed in

- Efficient LLM inference and serving / [Serving systems and request scheduling](https://mlsanity.com/inference/serving-systems-and-request-scheduling/)

## Builds on (most cited first, 29 of 29)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling](https://mlsanity.com/paper/2501.17811/) (2025)
- [Qwen2.5-Omni Technical Report](https://mlsanity.com/paper/2503.20215/) (2025)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023)
- [Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve](https://mlsanity.com/paper/2403.02310/) (2024)
- [MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback](https://mlsanity.com/paper/2309.10691/) (2023)
- [Fast Distributed Inference Serving for Large Language Models](https://mlsanity.com/paper/2305.05920/) (2023)
- [Vidur: A Large-Scale Simulation Framework For LLM Inference](https://mlsanity.com/paper/2405.05465/) (2024)
- [LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism](https://mlsanity.com/paper/2404.09526/) (2024)
- [DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency](https://mlsanity.com/paper/2408.00741/) (2024)
- [DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference](https://mlsanity.com/paper/2401.08671/) (2024)
- [BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems](https://mlsanity.com/paper/2401.17644/) (2024)
- [ServerlessLLM: Low-Latency Serverless Inference for Large Language Models](https://mlsanity.com/paper/2401.14351/) (2024)
- [Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads](https://mlsanity.com/paper/1901.05758/) (2019)
- [Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider](https://mlsanity.com/paper/2003.03423/) (2020)
- [MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving](https://mlsanity.com/paper/2404.02015/) (2024)
- [ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scale](https://mlsanity.com/paper/2303.14006/) (2023)
- [Characterization of Large Language Model Development in the Datacenter](https://mlsanity.com/paper/2403.07648/) (2024)
- [Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services](https://mlsanity.com/paper/2404.16283/) (2024)
- [Characterization and Prediction of Deep Learning Workloads in Large-Scale GPU Datacenters](https://mlsanity.com/paper/2109.01313/) (2021)
- [P/D-Serve: Serving Disaggregated Large Language Model at Scale](https://mlsanity.com/paper/2408.08147/) (2024)
- [semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storage](https://mlsanity.com/paper/2504.19867/) (2025)
- [Locality-aware Fair Scheduling in LLM Serving](https://mlsanity.com/paper/2501.14312/) (2025)
- [Characterizing Deep Learning Training Workloads on Alibaba-PAI](https://mlsanity.com/paper/1910.05930/) (2019)

## Built on by (most cited first, 29 of 29)

- [WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving](https://mlsanity.com/paper/2512.09472/) (2025)
- [Cornfigurator: Automated Planning for Any-to-Any Multimodal Model Serving](https://mlsanity.com/paper/2512.14098/) (2025)
- [Nitsum: Serving Tiered LLM Requests with Adaptive Tensor Parallelism](https://mlsanity.com/paper/2605.05467/) (2026)
- [Role-Based Fault Tolerance System for LLM RL Post-Training](https://mlsanity.com/paper/2512.22492/) (2025)
- [Simple is Better: Multiplication May Be All You Need for LLM Request Scheduling](https://mlsanity.com/paper/2603.15202/) (2026)
- [CRAFT: Fine-Grained Cost-Aware Expert Replication For Efficient Mixture-of-Experts Serving](https://mlsanity.com/paper/2603.28768/) (2026)
- [Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inference](https://mlsanity.com/paper/2604.09613/) (2026)
- [SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling](https://mlsanity.com/paper/2607.08565/) (2026)
- [MeAJOR Corpus: A Multi-Source Dataset for Phishing Email Detection](https://mlsanity.com/paper/2507.17978/) (2025)
- [TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inference](https://mlsanity.com/paper/2602.05145/) (2026)
- [Cornserve: A Distributed Serving System for Any-to-Any Multimodal Models](https://mlsanity.com/paper/2603.12118/) (2026)
- [The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Project](https://mlsanity.com/paper/2603.21354/) (2026)
- [Dual-Pool Token-Budget Routing for Cost-Efficient and Reliable LLM Serving](https://mlsanity.com/paper/2604.08075/) (2026)
- [ObjectCache: Layerwise Object-Storage Retrieval for KV Cache Reuse](https://mlsanity.com/paper/2605.22850/) (2026)
- [Diagnose Before You Compress: Prediction-Independent Bottleneck Witness Refinement for LLM Serving Traces](https://mlsanity.com/paper/2608.00423/) (2026)
- [Hydra: Phase-Aware Workload Characterization of LLM Inference across Edge SoC Generations, Backends, and Quantization Levels](https://mlsanity.com/paper/2608.25053/) (2026)
- [Hardware-Software Co-design for 3D-DRAM-based LLM Serving Accelerator](https://mlsanity.com/paper/2603.04797/) (2026)
- [Regulating Branch Parallelism in LLM Serving](https://mlsanity.com/paper/2605.06914/) (2026)
- [Geometry-Aware Online Scheduling for LLM Serving: From Theoretical Bound to System Practice](https://mlsanity.com/paper/2606.22327/) (2026)
- [ASAP: A Disaggregated and Asynchronous Inference System for MoE Prefill](https://mlsanity.com/paper/2606.22541/) (2026)
- [Energy-Aware Scheduling for Serverless LLM Serving on Shared GPUs](https://mlsanity.com/paper/2606.30391/) (2026)
- [NELSSA: A GPU-PNM Heterogeneous System for Mixed-Length LLM Serving via Length-based Request Placement](https://mlsanity.com/paper/2607.26633/) (2026)
- [InFactPlanner: Planning Sustainable Geo-Distributed LLM Data Centers](https://mlsanity.com/paper/2608.12915/) (2026)
- [A Year in LLM Serving: Workload Evolution, Caching and Load-Balancing](https://mlsanity.com/paper/2608.13573/) (2026)
- [Pallas: A Proactive KV Cache Migration Framework for LLM Inference in AI-RAN](https://mlsanity.com/paper/2608.16477/) (2026)
- [Token Latency Fairness: Performance Isolation for Multi-Tenant LLM Serving](https://mlsanity.com/paper/2609.18112/) (2026)
- [SARA: SLO-Aware Resource Allocation for Disaggregated Agentic LLM Services](https://mlsanity.com/paper/2609.26763/) (2026)
- [PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decoding](https://mlsanity.com/paper/2609.34555/) (2026)
- [Inference-Layer Security: Defending Against Adversarial Inference and Infrastructure Abuse](https://mlsanity.com/paper/2609.38239/) (2026)

## Abstract (arXiv)

With the widespread adoption of Large Language Models (LLMs), serving LLM inference requests has become an increasingly important task, attracting active research advancements. Practical workloads play an essential role in this process: they are critical for motivating and benchmarking serving techniques and systems. However, the existing understanding of real-world LLM serving workloads is limited due to the lack of a comprehensive workload characterization. Prior analyses remain insufficient in scale and scope, thus failing to fully capture intricate workload characteristics. In this paper, we fill the gap with an in-depth characterization of LLM serving workloads collected from our worldwide cloud inference serving service, covering not only language models but also emerging multimodal and reasoning models, and unveiling important new findings in each case. Moreover, based on our findings, we propose ServeGen, a principled framework for generating realistic LLM serving workloads by composing them on a per-client basis. A practical use case in production validates that ServeGen avoids 50% under-provisioning compared to naive workload generation, demonstrating ServeGen's advantage in performance benchmarking. ServeGen is available at https://github.com/alibaba/ServeGen.