mlsanityData to 14 Oct 2026

Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline

arXiv 2305.13144 ↗First posted 22 May 2023cs.CLcited by 86

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large language models (LLMs) have revolutionized the field of AI, demonstrating unprecedented capacity across various tasks. However, the inference process for LLMs comes with significant computational costs. In this paper, we propose an efficient LLM inference pipeline that harnesses the power of LLMs. Our approach begins by tapping into the potential of LLMs to accurately perceive and predict the response length with minimal overhead. By leveraging this information, we introduce an efficient sequence scheduling technique that groups queries with similar response lengths into micro-batches. We evaluate our approach on real-world instruction datasets using the LLaMA-based model, and our results demonstrate an impressive 86% improvement in inference throughput without compromising effectiveness. Notably, our method is orthogonal to other inference acceleration techniques, making it a valuable addition to many existing toolkits (e.g., FlashAttention, Quantization) for LLM inference.

Key facts. Cited by 86 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 41 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

25 of 41 references.

References · 25

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  8. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  10. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  11. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  12. 2022Efficiently Scaling Transformer Inferencecited by 539
13 more
  1. 2017Non-Autoregressive Neural Machine Translationcited by 463
  2. 2023Benchmarking Large Language Models for News Summarizationcited by 445
  3. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  4. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 203
  5. 2018Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinementcited by 195
  6. 2021Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Trainingcited by 96
  7. 2019FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flowcited by 89
  8. 2023Batch Prompting: Efficient Inference with Large Language Model APIscited by 85
  9. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 80
  10. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71
  11. 2019Fast Structured Decoding for Sequence Modelscited by 59
  12. 2019Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posteriorcited by 58
  13. 2020Iterative Refinement in the Continuous Space for Non-Autoregressive Neural Machine Translationcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 25; 2025: 33; 2026: 26.

86 papers.

Sort:

By area:

  1. 2024Token-Budget-Aware LLM Reasoningcited by 247
  2. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 112
  3. 2024On-Device Language Models: A Comprehensive Reviewcited by 104
  4. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systemscited by 87
  5. 2024Efficient LLM Scheduling by Learning to Rankcited by 85
  6. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Predictioncited by 66
  7. 2023Integrating Large Language Model for Improved Causal Discoverycited by 54
  8. 2024MegaAgent: A Large-Scale Autonomous LLM-based Multi-Agent System Without Predefined SOPscited by 47
  9. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  10. 2024Don't Stop Me Now: Embedding Based Scheduling for LLMscited by 35
  11. 2025OmniRouter: Budget and Performance Controllable Multi-LLM Routingcited by 35
  12. 2024M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneitycited by 31
38 more
  1. 2024Offline Energy-Optimal LLM Serving: Workload-Based Energy Models for LLM Inference on Heterogeneous Systemscited by 28
  2. 2025Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Servingcited by 23
  3. 2024SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Servingcited by 19
  4. 2025Online Scheduling for LLM Inference with KV Cache Constraintscited by 19
  5. 2025JITServe: SLO-aware LLM Serving with Imprecise Request Informationcited by 17
  6. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  7. 2024Slice-Level Scheduling for High Throughput and Load Balanced LLM Servingcited by 13
  8. 2024Aladdin: Joint Placement and Scaling for SLO-Aware LLM Servingcited by 11
  9. 2024Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Modelscited by 10
  10. 2024Fast Inference for Augmented Large Language Modelscited by 10
  11. 2025Adaptively Robust LLM Inference Optimization under Prediction Uncertaintycited by 10
  12. 2024Multi-Bin Batching for Increasing LLM Inference Throughputcited by 9
  13. 2025ELIS: Efficient LLM Iterative Scheduling System with Response Length Predictorcited by 9
  14. 2026Predicting LLM Output Length via Entropy-Guided Representationscited by 9
  15. 2025Mell: Memory-Efficient Large Language Model Serving via Multi-GPU KV Cache Managementcited by 8
  16. 2025Rethinking Key-Value Cache Compression Techniques for Large Language Model Servingcited by 8
  17. 2025An Empirical Study on Prompt Compression for Large Language Modelscited by 8
  18. 2026Scheduling LLM Inference with Uncertainty-Aware Output Length Predictionscited by 8
  19. 2025LLM Serving Optimization with Variable Prefill and Decode Lengthscited by 7
  20. 2023Generative AI Beyond LLMs: System Implications of Multi-Modal Generationcited by 6
  21. 2024EEG Emotion Copilot: Optimizing Lightweight LLMs for Emotional EEG Interpretation with Assisted Medical Record Generationcited by 6
  22. 2025PICE: A Semantic-Driven Progressive Inference System for LLM Serving in Cloud-Edge Networkscited by 6
  23. 2025Queueing, Predictions, and LLMs: Challenges and Open Problemscited by 5
  24. 2024Is the GPU Half-Empty or Half-Full? Practical Scheduling Techniques for LLMscited by 4
  25. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  26. 2025Semi-Clairvoyant Scheduling of Speculative Decoding Requests to Minimize LLM Inference Latencycited by 4
  27. 2025TimeBill: Time-Budgeted Inference for Large Language Modelscited by 4
  28. 2026Chimera: Latency- and Performance-Aware Multi-agent Serving for Heterogeneous LLMscited by 4
  29. 2025PARALLELPROMPT: Extracting Parallelism from Large Language Model Queriescited by 3
  30. 2025Correctness Forensics for Batch Speculative Decoding: Diagnosing the Ragged Tensor Problemcited by 3
  31. 2026Explainable Model Routing for Agentic Workflowscited by 3
  32. 2026Robust Length Prediction: A Perspective from Heavy-Tailed Prompt-Conditioned Distributionscited by 3
  33. 2024Performance Characterization of Expert Router for Scalable LLM Inferencecited by 2
  34. 2024ALISE: Accelerating Large Language Model Serving with Speculative Schedulingcited by 2
  35. 2025AccelGen: Heterogeneous SLO-Guaranteed High-Throughput LLM Inference Serving for Diverse Applicationscited by 2
  36. 2025High-Throughput LLM inference on Heterogeneous Clusterscited by 2
  37. 2025Scatter-Based Innovation Propagation in Large Language Models for Multi-Stage Process Adaptationcited by 2
  38. 2025Astrolabe: Balancing Load in LLM Serving with Randomized Prediction-Guided Schedulingcited by 2

50 most cited of 86 All 86 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024LLM Inference Unveiled: Survey and Roofline Model Insights
  2. 2023SparQ Attention: Bandwidth-Efficient LLM Inference
  3. 2023Efficient LLM Inference on CPUs
  4. 2024A Survey on Efficient Inference for Large Language Models
  5. 2026Prompt Compression in the Wild: Measuring Latency, Rate Adherence, and Quality for Faster LLM Inference
  6. 2025xLSTM 7B: A Recurrent LLM for Fast and Efficient Inference
  7. 2024Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM models
  8. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters