# BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems

arXiv: https://arxiv.org/abs/2401.17644 · HTML: https://mlsanity.com/paper/2401.17644/

## Key facts

- Cited by 87 papers in the map (data through 14 Oct 2026).
- Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving.
- Builds on 25 of its 59 references that are in the map.
- First posted on arXiv 31 Jan 2024.

## Placed in

- Efficient LLM inference and serving / [Serving systems and request scheduling](https://mlsanity.com/inference/serving-systems-and-request-scheduling/)

## Builds on (most cited first, 25 of 25)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [Fast Transformer Decoding: One Write-Head is All You Need](https://mlsanity.com/paper/1911.02150/) (2019)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023)
- [Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs](https://mlsanity.com/paper/2310.01801/) (2023)
- [SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification](https://mlsanity.com/paper/2305.09781/) (2023)
- [Atom: Low-bit Quantization for Efficient and Accurate LLM Serving](https://mlsanity.com/paper/2310.19102/) (2023)
- [The False Promise of Imitating Proprietary LLMs](https://mlsanity.com/paper/2305.15717/) (2023)
- [Fast Distributed Inference Serving for Large Language Models](https://mlsanity.com/paper/2305.05920/) (2023)
- [Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems](https://mlsanity.com/paper/2312.15234/) (2023)
- [SpotServe: Serving Generative Large Language Models on Preemptible Instances](https://mlsanity.com/paper/2311.15566/) (2023)
- [Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline](https://mlsanity.com/paper/2305.13144/) (2023)
- [FlashDecoding++: Faster Large Language Model Inference on GPUs](https://mlsanity.com/paper/2311.01282/) (2023)
- [Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider](https://mlsanity.com/paper/2003.03423/) (2020)
- [Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity](https://mlsanity.com/paper/2309.10285/) (2023)
- [Training and Serving System of Foundation Models: A Comprehensive Survey](https://mlsanity.com/paper/2401.02643/) (2024)
- [Failure Analysis of Big Cloud Service Providers Prior to and During Covid-19 Period](https://mlsanity.com/paper/2210.08006/) (2022)

## Built on by (most cited first, 30 of 87)

- [Llumnix: Dynamic Scheduling for Large Language Model Serving](https://mlsanity.com/paper/2406.03243/) (2024)
- [Energy Considerations of Large Language Model Inference and Efficiency Optimizations](https://mlsanity.com/paper/2504.17674/) (2025)
- [ServeGen: Workload Characterization and Generation of Large Language Model Serving in Production](https://mlsanity.com/paper/2505.09999/) (2025)
- [Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs](https://mlsanity.com/paper/2502.00722/) (2025)
- [SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Serving](https://mlsanity.com/paper/2408.05235/) (2024)
- [BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batching](https://mlsanity.com/paper/2411.16102/) (2024)
- [AdaSpec: Adaptive Speculative Decoding for Fast, SLO-Aware Large Language Model Serving](https://mlsanity.com/paper/2503.05096/) (2025)
- [HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Clouds](https://mlsanity.com/paper/2502.15524/) (2025)
- [Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantization](https://mlsanity.com/paper/2503.18599/) (2025)
- [Intelligence per Watt: Measuring Intelligence Efficiency of Local AI](https://mlsanity.com/paper/2511.07885/) (2025)
- [TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocity](https://mlsanity.com/paper/2512.03416/) (2025)
- [TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Scheduling](https://mlsanity.com/paper/2510.02758/) (2025)
- [ELIS: Efficient LLM Iterative Scheduling System with Response Length Predictor](https://mlsanity.com/paper/2505.09142/) (2025)
- [Janus: Disaggregating Attention and Experts for Scalable MoE Inference](https://mlsanity.com/paper/2512.13525/) (2025)
- [MorphServe: Efficient and Workload-Aware LLM Serving via Runtime Quantized Layer Swapping and KV Cache Resizing](https://mlsanity.com/paper/2506.02006/) (2025)
- [FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Serving](https://mlsanity.com/paper/2602.22593/) (2026)
- [SAGA: Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusters](https://mlsanity.com/paper/2605.00528/) (2026)
- [Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecture](https://mlsanity.com/paper/2505.11916/) (2025)
- [WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving](https://mlsanity.com/paper/2512.09472/) (2025)
- [Sustainable Carbon-Aware and Water-Efficient LLM Scheduling in Geo-Distributed Cloud Datacenters](https://mlsanity.com/paper/2505.23554/) (2025)
- [TokTier: Exact Stateful CPU+GPU Tokenization for Agentic LLM Serving](https://mlsanity.com/paper/2607.29678/) (2026)
- [RAP: Runtime Adaptive Pruning for LLM Inference](https://mlsanity.com/paper/2505.17138/) (2025)
- [Towards Resiliency in Large Language Model Serving with KevlarFlow](https://mlsanity.com/paper/2601.22438/) (2026)
- [PipeLive: Efficient Live In-place Pipeline Parallelism Reconfiguration for Dynamic LLM Serving](https://mlsanity.com/paper/2604.12171/) (2026)
- [GhostServe: A Lightweight Checkpointing System in the Shadow for Fault-Tolerant LLM Serving](https://mlsanity.com/paper/2605.00831/) (2026)
- [Nitsum: Serving Tiered LLM Requests with Adaptive Tensor Parallelism](https://mlsanity.com/paper/2605.05467/) (2026)
- [Green SARC: Predictive Cost and Carbon Governance for Agentic AI Systems](https://mlsanity.com/paper/2606.15954/) (2026)
- [Efficient LLM Serving on Hybrid Real-time and Best-effort Requests](https://mlsanity.com/paper/2504.09590/) (2025)
- [RAGDoll: Efficient Offloading-based Online RAG System on a Single GPU](https://mlsanity.com/paper/2504.15302/) (2025)
- [AGFT: An Adaptive GPU Frequency Tuner for Real-Time LLM Inference Optimization](https://mlsanity.com/paper/2508.01744/) (2025)

## Abstract (arXiv)

Serving systems for Large Language Models (LLMs) are often optimized to improve quality of service (QoS) and throughput. However, due to the lack of open-source LLM serving workloads, these systems are frequently evaluated under unrealistic workload assumptions. Consequently, performance may degrade when systems are deployed in real-world scenarios. This work presents BurstGPT, an LLM serving workload with 10.31 million traces from regional Azure OpenAI GPT services over 213 days. BurstGPT captures LLM serving characteristics from user, model and system perspectives: (1) User request concurrency: burstiness variations of requests in Azure OpenAI GPT services, revealing diversified concurrency patterns in different services and model types. (2) User conversation patterns: counts and intervals within conversations for service optimizations. (3) Model response lengths: auto-regressive serving processes of GPT models, showing statistical relations between requests and their responses. (4) System response failures: failures of conversation and API services, showing intensive resource needs and limited availability of LLM services in Azure. The details of the characteristics can serve multiple purposes in LLM serving optimizations, such as system evaluation and trace provisioning. In our demo evaluation with BurstGPT, frequent variations in BurstGPT reveal declines in efficiency, stability, or reliability in realistic LLM serving. We identify that the generalization of KV cache management, scheduling and disaggregation optimizations can be improved under realistic workload evaluations. BurstGPT is publicly available now at https://github.com/HPMLL/BurstGPT and is widely used to develop prototypes of LLM serving frameworks in the industry.