A Year in LLM Serving: Workload Evolution, Caching and Load-Balancing
On the map: Efficient LLM inference and serving → Serving systems and request scheduling
Abstract (arXiv)
Large Language Model (LLM) serving has become a critical cloud workload, and realistic traces are essential for motivating and benchmarking serving systems. However, existing LLM serving workload studies remain limited in scale and scope. They often observe short time periods and provide limited visibility into how users interact with models in production. As a result, they do not fully capture how LLM serving workloads evolve over time or how user-model interactions shape production traffic. In this work, we further the understanding of real-world LLM serving workloads through both a global characterization and a longitudinal study of a one-year production trace from CompanyX. Unlike prior studies, our trace captures full production behavior across many models and users, including both popular and long-tail models. We analyze the workload from aggregate, temporal, model-level, and user-level perspectives, revealing workload evolution and user-model structure that are typically hidden behind aggregate views. To support future research, we publicly release the full one-year trace, enabling downstream studies of production behavior without relying on sampled or synthetically generated workloads. The trace is available at https://github.com/HarvardMadSys/chutes_workload.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 14 of its 30 references that are in the map. First posted on arXiv 3 Jul 2026.
Builds on
14 of 30 references.
Closest ancestors · 5
Same direction, newest first
- 2025ServeGen: Workload Characterization and Generation of Large Language Model Serving in Production15 May
- 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
- 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
- 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems31 Jan
- 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
Other references · 9
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
- 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
- 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
- 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
- 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 223
- 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 126
- 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Providercited by 41
- 2017MITHRIL: Mining Sporadic Associations for Cache Prefetchingcited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026FineServe: A Fine-Grained Dataset and Characterization of Global LLM Serving Workloads
- 2026LLM Serving in the Wild: An Empirical Study of Frameworks, Methods, and System Designs
- 2026OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestration
- 2024ScaleLLM: A Resource-Frugal LLM Serving Framework by Optimizing End-to-End Efficiency
- 2025Unlock the Potential of Fine-grained LLM Serving via Dynamic Module Scaling
- 2024The CAP Principle for LLM Serving: A Survey of Long-Context Large Language Model Serving
- 2025SageServe: Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scaling
- 2026Autopoiesis: A Self-Evolving System Paradigm for LLM Serving Under Runtime Dynamics