mlsanityData to 7 Oct 2026

BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batching

arXiv 2411.16102 ↗First posted 25 Nov 2024cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

Offline batch inference, which leverages the flexibility of request batching to achieve higher throughput and lower costs, is becoming more popular for latency-insensitive applications. Meanwhile, recent progress in model capability and modality makes requests more diverse in compute and memory demands, creating unique opportunities for throughput improvement by resource overlapping. However, a request schedule that maximizes resource overlapping can conflict with the schedule that maximizes prefix sharing, a widely-used performance optimization, causing sub-optimal inference throughput. We present BlendServe, a system that maximizes resource utilization of offline batch inference by combining the benefits of resource overlapping and prefix sharing using a resource-aware prefix tree. BlendServe exploits the relaxed latency requirements in offline batch inference to reorder and overlap requests with varied resource demands while ensuring high prefix sharing. We evaluate BlendServe on a variety of synthetic multi-modal workloads and show that it provides up to 1.44× throughput boost compared to widely-used industry standards, vLLM and SGLang.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 45 of its 92 references that are in the map. First posted on arXiv 25 Nov 2024.

Builds on

45 of 92 references.

References · 45

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2023Qwen Technical Reportcited by 3,273
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  8. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  10. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  12. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
33 more
  1. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  2. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  3. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  4. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  5. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  6. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  7. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
  8. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  9. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  10. 2024OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generationcited by 332
  11. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  12. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  13. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  14. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
  15. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 239
  16. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  17. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 196
  18. 2023Fast Distributed Inference Serving for Large Language Modelscited by 143
  19. 2024O1 Replication Journey: A Strategic Progress Report -- Part 1cited by 140
  20. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  21. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 106
  22. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 96
  23. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughputcited by 85
  24. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systemscited by 84
  25. 2024Efficient LLM Scheduling by Learning to Rankcited by 83
  26. 2023Fairness in Serving Large Language Modelscited by 77
  27. 2024Hydragen: High-Throughput LLM Inference with Shared Prefixescited by 71
  28. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Modelscited by 69
  29. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 68
  30. 2020Multi-Head Attention: Collaborate Instead of Concatenatecited by 56
  31. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelinescited by 49
  32. 2024MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Servingcited by 49
  33. 2024MIO: A Foundation Model on Multimodal Tokenscited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 14; 2026: 5.

19 papers.

Sort:

By area:

  1. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 119
  2. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUscited by 21
  3. 2025HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-locationcited by 10
  4. 2025Accelerating Large-Scale Reasoning Model Inference with Sparse Self-Speculative Decodingcited by 9
  5. 2025MoE-Gen: High-Throughput MoE Inference on a Single GPU with Module-Based Batchingcited by 6
  6. 2025Cascadia: An Efficient Cascade Serving System for Large Language Modelscited by 5
  7. 2025Optimal Scheduling Algorithms for LLM Inference: Theory and Practicecited by 5
  8. 2025TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Servingcited by 5
  9. 2025CoDec: Prefix-Shared Decoding Kernel for LLMscited by 4
  10. 2026PROBE: Co-Balancing Computation and Communication in MoE Inference via Real-Time Predictive Prefetchingcited by 4
  11. 2025Echo: Efficient Co-Scheduling of Hybrid Online-Offline Tasks for Large Language Model Servingcited by 3
  12. 2025ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Cachingcited by 3
7 more
  1. 2025Efficient LLM Serving on Hybrid Real-time and Best-effort Requestscited by 2
  2. 2026OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestrationcited by 2
  3. 2025PerCache: Predictive Hierarchical Cache for RAG Applications on Mobile Devicescited by 1
  4. 2026MoE-Prefill: Zero Redundancy Overheads in MoE Prefill Servingcited by 1
  5. 2025Improving the End-to-End Efficiency of Offline Inference for Multi-LLM Applications Based on Sampling and Simulationcited by 0
  6. 2026PipeMax: Enhancing Offline LLM Inference on Commodity GPU Serverscited by 0
  7. 2026HexAGenT: Efficient Agentic LLM Serving via Workflow- and Heterogeneity-Aware Schedulingcited by 0