mlsanityData to 7 Oct 2026

DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference

arXiv 2401.08671 ↗First posted 9 Jan 2024cs.PF cs.LGcited by 106

Not on the map. Why?

Cited by 62 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The deployment and scaling of large language models (LLMs) have become critical as they permeate various applications, demanding high-throughput and low-latency serving systems. Existing frameworks struggle to balance these requirements, especially for workloads with long prompts. This paper introduces DeepSpeed-FastGen, a system that employs Dynamic SplitFuse, a novel prompt and generation composition strategy, to deliver up to 2.3x higher effective throughput, 2x lower latency on average, and up to 3.7x lower (token-level) tail latency, compared to state-of-the-art systems like vLLM. We leverage a synergistic combination of DeepSpeed-MII and DeepSpeed-Inference to provide an efficient and easy-to-use serving system for LLMs. DeepSpeed-FastGen's advanced implementation supports a range of models and offers both non-persistent and persistent deployment options, catering to diverse user scenarios from interactive sessions to long-running applications. We present a detailed benchmarking methodology, analyze the performance through latency-throughput curves, and investigate scalability via load balancing. Our evaluations demonstrate substantial improvements in throughput and latency across various models and hardware configurations. We discuss our roadmap for future enhancements, including broader model support and new hardware backends. The DeepSpeed-FastGen code is readily available for community engagement and contribution.

Key facts. Cited by 106 papers in the map (data through 7 Oct 2026). Builds on 6 of its 12 references that are in the map. First posted on arXiv 9 Jan 2024.

Builds on

6 of 12 references.

References · 6

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  5. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 239
  6. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 36; 2025: 43; 2026: 27.

106 papers.

Sort:

By area:

  1. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  2. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  3. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  4. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 110
  5. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  6. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 96
  7. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 93
  8. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughputcited by 85
  9. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  10. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compressioncited by 84
  11. 2024Efficiently Scaling LLM Reasoning with Certaindexcited by 56
  12. 2024LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategycited by 50
38 more
  1. 2025KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environmentscited by 49
  2. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  3. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 46
  4. 2024ProMoE: Fast MoE-based LLM Serving using Proactive Cachingcited by 38
  5. 2024Fast State Restoration in LLM Serving with HCachecited by 32
  6. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  7. 2024BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batchingcited by 31
  8. 2025ServeGen: Workload Characterization and Generation of Large Language Model Serving in Productioncited by 29
  9. 2025Jenga: Effective Memory Management for Serving LLM with Heterogeneitycited by 24
  10. 2025Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipelinecited by 20
  11. 2025Past-Future Scheduler for LLM Serving under SLA Guaranteescited by 20
  12. 2026FASTER: Rethinking Real-Time Flow VLAscited by 20
  13. 2024InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inferencecited by 19
  14. 2024BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batchingcited by 19
  15. 2024SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformationcited by 18
  16. 2024MixLLM: LLM Quantization with Global Mixed-precision between Output-features and Highly-efficient System Designcited by 17
  17. 2024LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Managementcited by 16
  18. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  19. 2024L4Q: Parameter Efficient Quantization-Aware Fine-Tuning on Large Language Modelscited by 15
  20. 2025Seesaw: High-throughput LLM Inference via Model Re-shardingcited by 15
  21. 2025semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storagecited by 15
  22. 2025CAOTE: KV Cache Selection for LLMs via Attention Output Error-Based Token Evictioncited by 13
  23. 2024Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM modelscited by 12
  24. 2024KunServe: Parameter-centric Memory Management for Efficient Memory Overloading Handling in LLM Servingcited by 10
  25. 2025Rectified Sparse Attentioncited by 10
  26. 2025Decoder-Hybrid-Decoder Architecture for Efficient Reasoning with Long Generationcited by 10
  27. 2024KVDirect: Distributed Disaggregated LLM Inferencecited by 8
  28. 2026ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compressioncited by 8
  29. 2025ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelismcited by 7
  30. 2024COMET: Towards Partical W4A4KV4 LLMs Servingcited by 6
  31. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Servingcited by 6
  32. 2025EAIRA: Establishing a Methodology for Evaluating AI Models as Scientific Research Assistantscited by 5
  33. 2025A Novel Hat-Shaped Device-Cloud Collaborative Inference Framework for Large Language Modelscited by 5
  34. 2025Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecturecited by 5
  35. 2025Optimal Scheduling Algorithms for LLM Inference: Theory and Practicecited by 5
  36. 2025TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Servingcited by 5
  37. 2025A Cost-Effective Near-Storage Processing Solution for Offline Inference of Long-Context LLMscited by 4
  38. 2025Ascendra: Dynamic Request Prioritization for Efficient LLM Servingcited by 4

50 most cited of 106 All 106 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Comparative Analysis of Large Language Model Inference Serving Systems: A Performance Study of vLLM and HuggingFace TGI
  2. 2023Splitwise: Efficient generative LLM inference using phase splitting
  3. 2026Prompt Compression in the Wild: Measuring Latency, Rate Adherence, and Quality for Faster LLM Inference
  4. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective
  5. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems
  6. 2025FastMTP: Accelerating LLM Inference with Enhanced Multi-Token Prediction
  7. 2026Parallelization Strategies for Dense LLM Deployment: Navigating Through Application-Specific Tradeoffs and Bottlenecks
  8. 2025AccLLM: Accelerating Long-Context LLM Inference Via Algorithm-Hardware Co-Design