mlsanityData to 7 Oct 2026

S3: Increasing GPU Utilization during Generative Inference for Higher Throughput

arXiv 2306.06000 ↗First posted 9 Jun 2023cs.AR cs.AIcited by 81

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Generating texts with a large language model (LLM) consumes massive amounts of memory. Apart from the already-large model parameters, the key/value (KV) cache that holds information about previous tokens in a sequence can grow to be even larger than the model itself. This problem is exacerbated in one of the current LLM serving frameworks which reserves the maximum sequence length of memory for the KV cache to guarantee generating a complete sequence as they do not know the output sequence length. This restricts us to use a smaller batch size leading to lower GPU utilization and above all, lower throughput. We argue that designing a system with a priori knowledge of the output sequence can mitigate this problem. To this end, we propose S3, which predicts the output sequence length, schedules generation queries based on the prediction to increase device resource utilization and throughput, and handle mispredictions. Our proposed method achieves 6.49× throughput over those systems that assume the worst case for the output sequence length.

Key facts. Cited by 81 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 28 of its 48 references that are in the map. First posted on arXiv 9 Jun 2023.

Builds on

28 of 48 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  8. 2020Rethinking Attention with Performerscited by 1,584
  9. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  10. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  11. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  12. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
16 more
  1. 2022Efficiently Scaling Transformer Inferencecited by 529
  2. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  3. 2021Post-Training Quantization for Vision Transformercited by 274
  4. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  5. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 228
  6. 2019MLPerf Inference Benchmarkcited by 195
  7. 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
  8. 2022Can Foundation Models Wrangle Your Data?cited by 173
  9. 2023Fast Distributed Inference Serving for Large Language Modelscited by 143
  10. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 119
  11. 2017TensorFlow-Serving: Flexible, High-Performance ML Servingcited by 85
  12. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71
  13. 2020DeepRecSys: A System for Optimizing End-To-End At-scale Neural Recommendation Inferencecited by 54
  14. 2020Sub-Linear Memory: How to Make Performers SLiMcited by 18
  15. 2018DLHub: Model and Data Serving for Sciencecited by 17
  16. 2023MP-Rec: Hardware-Software Co-Design to Enable Multi-Path Recommendationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 25; 2025: 32; 2026: 22.

81 papers.

Sort:

By area:

  1. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  2. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  3. 2023Efficient Large Language Models: A Surveycited by 183
  4. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 106
  5. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  6. 2024Efficient LLM Scheduling by Learning to Rankcited by 83
  7. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partitioncited by 81
  8. 2024Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methodscited by 80
  9. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Predictioncited by 64
  10. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  11. 2024MegaAgent: A Large-Scale Autonomous LLM-based Multi-Agent System Without Predefined SOPscited by 45
  12. 2025OmniRouter: Budget and Performance Controllable Multi-LLM Routingcited by 35
38 more
  1. 2024Don't Stop Me Now: Embedding Based Scheduling for LLMscited by 34
  2. 2025Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inferencecited by 32
  3. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  4. 2024M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneitycited by 30
  5. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 21
  6. 2025Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Servingcited by 21
  7. 2024AsymKV: Enabling 1-Bit Quantization of KV Cache with Layer-Wise Asymmetric Quantization Configurationscited by 19
  8. 2024SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Servingcited by 18
  9. 2025Online Scheduling for LLM Inference with KV Cache Constraintscited by 17
  10. 2025JITServe: SLO-aware LLM Serving with Imprecise Request Informationcited by 17
  11. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  12. 2024Slice-Level Scheduling for High Throughput and Load Balanced LLM Servingcited by 13
  13. 2025MoQAE: Mixed-Precision Quantization for Long-Context LLM Inference via Mixture of Quantization-Aware Expertscited by 12
  14. 2024Fast Inference for Augmented Large Language Modelscited by 10
  15. 2025HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-locationcited by 10
  16. 2024TPI-LLM: Serving 70B-scale LLMs Efficiently on Low-resource Edge Devicescited by 9
  17. 2024Multi-Bin Batching for Increasing LLM Inference Throughputcited by 9
  18. 2026Predicting LLM Output Length via Entropy-Guided Representationscited by 9
  19. 2025ELIS: Efficient LLM Iterative Scheduling System with Response Length Predictorcited by 8
  20. 2025SQuat: Subspace-orthogonal KV Cache Quantizationcited by 7
  21. 2025Quality-of-Service Aware LLM Routing for Edge Computing with Multiple Expertscited by 7
  22. 2026Scheduling LLM Inference with Uncertainty-Aware Output Length Predictionscited by 7
  23. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Servingcited by 6
  24. 2025Scorpio: Serving Right Requests at the Right Time for Heterogeneous SLOs in LLM Inferencecited by 6
  25. 2024Towards Pareto Optimal Throughput in Small Language Model Servingcited by 5
  26. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Servingcited by 5
  27. 2026TRIAGE: Evaluating Prospective Metacognitive Control in LLMs under Resource Constraintscited by 5
  28. 2024Is the GPU Half-Empty or Half-Full? Practical Scheduling Techniques for LLMscited by 4
  29. 2025Queueing, Predictions, and LLMs: Challenges and Open Problemscited by 4
  30. 2025TimeBill: Time-Budgeted Inference for Large Language Modelscited by 4
  31. 2024Ensuring Fair LLM Serving Amid Diverse Applicationscited by 3
  32. 2025Echo: Efficient Co-Scheduling of Hybrid Online-Offline Tasks for Large Language Model Servingcited by 3
  33. 2026FlowPrefill: Decoupling Preemption from Prefill Scheduling Granularity to Mitigate Head-of-Line Blocking in LLM Servingcited by 3
  34. 2026Robust Length Prediction: A Perspective from Heavy-Tailed Prompt-Conditioned Distributionscited by 3
  35. 2024ALISE: Accelerating Large Language Model Serving with Speculative Schedulingcited by 2
  36. 2025AccelGen: Heterogeneous SLO-Guaranteed High-Throughput LLM Inference Serving for Diverse Applicationscited by 2
  37. 2025High-Throughput LLM inference on Heterogeneous Clusterscited by 2
  38. 2025Astrolabe: Balancing Load in LLM Serving with Randomized Prediction-Guided Schedulingcited by 2

50 most cited of 81 All 81 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
  2. 2024XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference
  3. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  4. 2025TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization
  5. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management
  6. 2024RefreshKV: Updating Small KV Cache During Long-form Generation
  7. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption
  8. 2025SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs