mlsanityData to 7 Oct 2026

DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale

arXiv 2207.00032 ↗First posted 30 Jun 2022cs.LG cs.DC cs.PFcited by 403

Not on the map. Why?

Cited by 126 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The past several years have witnessed the success of transformer-based models, and their scale and application scenarios continue to grow aggressively. The current landscape of transformer models is increasingly diverse: the model size varies drastically with the largest being of hundred-billion parameters; the model characteristics differ due to the sparsity introduced by the Mixture-of-Experts; the target application scenarios can be latency-critical or throughput-oriented; the deployment hardware could be single- or multi-GPU systems with different types of memory and storage, etc. With such increasing diversity and the fast-evolving pace of transformer models, designing a highly performant and efficient inference system is extremely challenging. In this paper, we present DeepSpeed Inference, a comprehensive system solution for transformer model inference to address the above-mentioned challenges. DeepSpeed Inference consists of (1) a multi-GPU inference solution to minimize latency while maximizing the throughput of both dense and sparse transformer models when they fit in aggregate GPU memory, and (2) a heterogeneous inference solution that leverages CPU and NVMe memory in addition to the GPU memory and compute to enable high inference throughput with large models which do not fit in aggregate GPU memory. DeepSpeed Inference reduces latency by up to 7.3X over the state-of-the-art for latency-oriented scenarios and increases throughput by over 1.5x for throughput-oriented scenarios. Moreover, it enables trillion parameter scale inference under real-time latency constraints by leveraging hundreds of GPUs, an unprecedented scale for inference. It can inference 25x larger models than with GPU-only solutions, while delivering a high throughput of 84 TFLOPS (over 50% of A6000 peak).

Key facts. Cited by 403 papers in the map (data through 7 Oct 2026). Builds on 18 of its 39 references that are in the map. First posted on arXiv 30 Jun 2022.

Builds on

18 of 39 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  8. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  9. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  10. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  11. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  12. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
6 more
  1. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  2. 2018PipeDream: Fast and Efficient Pipeline Parallel DNN Trainingcited by 116
  3. 2020Data Movement Is All You Need: A Case Study on Optimizing Transformerscited by 108
  4. 2021The Efficiency Misnomercited by 98
  5. 2021Scalable and Efficient MoE Training for Multitask Multilingual Modelscited by 76
  6. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 61; 2024: 118; 2025: 131; 2026: 90.

403 papers.

Sort:

By area:

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  3. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  4. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  5. 2022Efficiently Scaling Transformer Inferencecited by 529
  6. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  7. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 295
  8. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  9. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 253
  10. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  11. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  12. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 212
38 more
  1. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  2. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  3. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  4. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  5. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  6. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 170
  7. 2023ChipNeMo: Domain-Adapted LLMs for Chip Designcited by 168
  8. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  9. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  10. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inferencecited by 148
  11. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  12. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 146
  13. 2023Fast Distributed Inference Serving for Large Language Modelscited by 143
  14. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  15. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  16. 2023Chatlaw: A Multi-Agent Legal Assistant based on a Role-Aligned Mixture-of-Experts Architecturecited by 119
  17. 2023RPTQ: Reorder-based Post-training Quantization for Large Language Modelscited by 112
  18. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 112
  19. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decodingcited by 103
  20. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inferencecited by 101
  21. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  22. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 98
  23. 2023Algorithm of Thoughts: Enhancing Exploration of Ideas in Large Language Modelscited by 92
  24. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 91
  25. 2023Fast Inference of Mixture-of-Experts Language Models with Offloadingcited by 88
  26. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decodingcited by 87
  27. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipelinecited by 86
  28. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instancescited by 83
  29. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  30. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partitioncited by 81
  31. 2023Punica: Multi-Tenant LoRA Servingcited by 80
  32. 2024Frame-Voyager: Learning to Query Frames for Video Large Language Modelscited by 80
  33. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 78
  34. 2023Fairness in Serving Large Language Modelscited by 77
  35. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73
  36. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 71
  37. 2024Hydragen: High-Throughput LLM Inference with Shared Prefixescited by 71
  38. 2023FlashDecoding++: Faster Large Language Model Inference on GPUscited by 70

50 most cited of 403 All 403 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Parallel Track Transformers: Enabling Fast GPU Inference with Reduced Synchronization
  2. 2025CoFormer: Collaborating with Heterogeneous Edge Devices for Scalable Transformer Inference
  3. 2024Ultra-Sparse Memory Network
  4. 2026LEAP: Layer-wise Exit-Aware Pretraining for Efficient Transformer Inference
  5. 2025Optimizing Inference in Transformer-Based Models: A Multi-Method Benchmark
  6. 2025EdgeFlex-Transformer: Transformer Inference for Edge Devices
  7. 2026FlashSVD v1.5: Making Low-Rank Transformers Inference Actually Fast
  8. 2025EcoSpa: Efficient Transformer Training with Coupled Sparsity