mlsanityData to 14 Oct 2026

BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems

arXiv 2401.17644 ↗First posted 31 Jan 2024cs.DC cs.PFcited by 87

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Serving systems for Large Language Models (LLMs) are often optimized to improve quality of service (QoS) and throughput. However, due to the lack of open-source LLM serving workloads, these systems are frequently evaluated under unrealistic workload assumptions. Consequently, performance may degrade when systems are deployed in real-world scenarios. This work presents BurstGPT, an LLM serving workload with 10.31 million traces from regional Azure OpenAI GPT services over 213 days. BurstGPT captures LLM serving characteristics from user, model and system perspectives: (1) User request concurrency: burstiness variations of requests in Azure OpenAI GPT services, revealing diversified concurrency patterns in different services and model types. (2) User conversation patterns: counts and intervals within conversations for service optimizations. (3) Model response lengths: auto-regressive serving processes of GPT models, showing statistical relations between requests and their responses. (4) System response failures: failures of conversation and API services, showing intensive resource needs and limited availability of LLM services in Azure. The details of the characteristics can serve multiple purposes in LLM serving optimizations, such as system evaluation and trace provisioning. In our demo evaluation with BurstGPT, frequent variations in BurstGPT reveal declines in efficiency, stability, or reliability in realistic LLM serving. We identify that the generalization of KV cache management, scheduling and disaggregation optimizations can be improved under realistic workload evaluations. BurstGPT is publicly available now at https://github.com/HPMLL/BurstGPT and is widely used to develop prototypes of LLM serving frameworks in the industry.

Key facts. Cited by 87 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 59 references that are in the map. First posted on arXiv 31 Jan 2024.

Builds on

25 of 59 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  2. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems23 Dec
  3. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instances27 Nov
  4. 2023FlashDecoding++: Faster Large Language Model Inference on GPUs2 Nov
  5. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline22 May
  6. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 19

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  7. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  9. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  10. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  11. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  12. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
7 more
  1. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 365
  2. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  3. 2023The False Promise of Imitating Proprietary LLMscited by 263
  4. 2020Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Providercited by 66
  5. 2023Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsitycited by 17
  6. 2024Training and Serving System of Foundation Models: A Comprehensive Surveycited by 11
  7. 2022Failure Analysis of Big Cloud Service Providers Prior to and During Covid-19 Periodcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 29; 2026: 54.

87 papers.

Sort:

By area:

  1. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 174
  2. 2025Energy Considerations of Large Language Model Inference and Efficiency Optimizationscited by 55
  3. 2025ServeGen: Workload Characterization and Generation of Large Language Model Serving in Productioncited by 29
  4. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUscited by 22
  5. 2024SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Servingcited by 19
  6. 2024BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batchingcited by 19
  7. 2025AdaSpec: Adaptive Speculative Decoding for Fast, SLO-Aware Large Language Model Servingcited by 18
  8. 2025HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Cloudscited by 14
  9. 2025Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantizationcited by 14
  10. 2025Intelligence per Watt: Measuring Intelligence Efficiency of Local AIcited by 14
  11. 2025TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocitycited by 12
  12. 2025TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Schedulingcited by 10
38 more
  1. 2025ELIS: Efficient LLM Iterative Scheduling System with Response Length Predictorcited by 9
  2. 2025Janus: Disaggregating Attention and Experts for Scalable MoE Inferencecited by 8
  3. 2025MorphServe: Efficient and Workload-Aware LLM Serving via Runtime Quantized Layer Swapping and KV Cache Resizingcited by 6
  4. 2026FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Servingcited by 6
  5. 2026SAGA: Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusterscited by 6
  6. 2025Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecturecited by 5
  7. 2025WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Servingcited by 5
  8. 2025Sustainable Carbon-Aware and Water-Efficient LLM Scheduling in Geo-Distributed Cloud Datacenterscited by 4
  9. 2026TokTier: Exact Stateful CPU+GPU Tokenization for Agentic LLM Servingcited by 4
  10. 2025RAP: Runtime Adaptive Pruning for LLM Inferencecited by 3
  11. 2026Towards Resiliency in Large Language Model Serving with KevlarFlowcited by 3
  12. 2026PipeLive: Efficient Live In-place Pipeline Parallelism Reconfiguration for Dynamic LLM Servingcited by 3
  13. 2026GhostServe: A Lightweight Checkpointing System in the Shadow for Fault-Tolerant LLM Servingcited by 3
  14. 2026Nitsum: Serving Tiered LLM Requests with Adaptive Tensor Parallelismcited by 3
  15. 2026Green SARC: Predictive Cost and Carbon Governance for Agentic AI Systemscited by 3
  16. 2025Efficient LLM Serving on Hybrid Real-time and Best-effort Requestscited by 2
  17. 2025RAGDoll: Efficient Offloading-based Online RAG System on a Single GPUcited by 2
  18. 2025AGFT: An Adaptive GPU Frequency Tuner for Real-Time LLM Inference Optimizationcited by 2
  19. 2025Astrolabe: Balancing Load in LLM Serving with Randomized Prediction-Guided Schedulingcited by 2
  20. 2025Rethinking Caching for LLM Serving Systems: Beyond Traditional Heuristicscited by 2
  21. 2025Understanding and Improving Communication Performance in Multi-node LLM Inferencecited by 2
  22. 2025RAGPulse: An Open-Source RAG Workload Trace to Optimize RAG Serving Systemscited by 2
  23. 2025RepetitionCurse: Measuring and Understanding Router Imbalance in Mixture-of-Experts LLMs under DoS Stresscited by 2
  24. 2026Analytical Provisioning for Attention-FFN Disaggregated LLM Serving under Stochastic Workloadscited by 2
  25. 2026Rethinking Latency Denial-of-Service: Attacking the LLM Serving Framework, Not the Modelcited by 2
  26. 2026Daily and Weekly Periodicity in Large Language Model Performance and Its Implications for Researchcited by 2
  27. 2026Simple is Better: Multiplication May Be All You Need for LLM Request Schedulingcited by 2
  28. 2026StepCache: Step-Level Reuse with Lightweight Verification and Selective Patching for LLM Servingcited by 2
  29. 2026Foundry: Template-Based CUDA Graph Context Materialization for Fast LLM Serving Cold Startcited by 2
  30. 2026Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inferencecited by 2
  31. 2026Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUscited by 2
  32. 2026TreeWY: Speculative Verification for Gated DeltaNet Hybridscited by 2
  33. 2025ServerlessLoRA: Enabling Low-Latency Serverless Multi-LoRA Servingcited by 1
  34. 2026TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inferencecited by 1
  35. 2026A Picture of Agentic Searchcited by 1
  36. 2026Multi-Layer Scheduling for MoE-Based LLM Reasoningcited by 1
  37. 2026Adaptive Multi-Objective Tiered Storage Configuration for KV Cache in LLM Servicecited by 1
  38. 2026Autopoiesis: A Self-Evolving System Paradigm for LLM Serving Under Runtime Dynamicscited by 1

50 most cited of 87 All 87 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026LLM Serving in the Wild: An Empirical Study of Frameworks, Methods, and System Designs
  2. 2024ScaleLLM: A Resource-Frugal LLM Serving Framework by Optimizing End-to-End Efficiency
  3. 2026OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestration
  4. 2025BucketServe: Bucket-Based Dynamic Batching for Smart and Efficient LLM Inference Serving
  5. 2025Niyama : Breaking the Silos of LLM Inference Serving
  6. 2025Recursive Offloading for LLM Serving in Multi-tier Networks
  7. 2024Queue management for slo-oriented large language model serving
  8. 2025JITServe: SLO-aware LLM Serving with Imprecise Request Information