mlsanityData to 7 Oct 2026

LLM Inference Under Bursty Workload Distribution: Modifying the WAIT Algorithm

arXiv 2608.06135 ↗First posted 6 Aug 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) such as ChatGPT and Claude are widely used for information retrieval and problem-solving. Recent work has focused on improving scheduling algorithms to boost throughput while maintaining low latency. However, these approaches often assume Poisson request arrivals with constant rates - an assumption that fails to reflect the inherently bursty and dynamic nature of real-world traffic. We propose a lightweight extension to the state-of-the-art WAIT algorithm [1], which adapts to time-varying arrival rates without prior traffic knowledge. The proposed algorithm performs online estimation of request intensity based on observed interarrival times. Using Markov Modulated Poisson Process (MMPP)-based synthetic workloads with diverse request types, we conduct a simulation-based evaluation demonstrating that the proposed method achieves higher throughput than Sarathi-Serve [2], ORCA [3], and vLLM [4] in the evaluated low arrival-rate shift scenarios while maintaining comparable latency.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 20 of its 45 references that are in the map. First posted on arXiv 6 Aug 2026.

Builds on

20 of 45 references.

References · 20

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  6. 2023PaLM 2 Technical Reportcited by 1,154
  7. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  8. 2020Efficient Transformers: A Surveycited by 784
  9. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  10. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  11. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  12. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
8 more
  1. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systemscited by 84
  2. 2024Efficient LLM Scheduling by Learning to Rankcited by 83
  3. 2023Punica: Multi-Tenant LoRA Servingcited by 80
  4. 2015The Sample Complexity of Revenue Maximizationcited by 48
  5. 2025Throughput-Optimal Scheduling Algorithms for LLM Inference and AI Agentscited by 17
  6. 2025Optimizing LLM Inference: Fluid-Guided Online Scheduling with Memory Constraintscited by 16
  7. 2025ELIS: Efficient LLM Iterative Scheduling System with Response Length Predictorcited by 8
  8. 2025Queueing, Predictions, and LLMs: Challenges and Open Problemscited by 4

Built on by

Not cited yet.