mlsanityData to 7 Oct 2026

TurboTransformers: An Efficient GPU Serving System For Transformer Models

arXiv 2010.05680 ↗First posted 9 Oct 2020cs.DC cs.AI cs.LGcited by 71

Not on the map. Why?

Cited by 31 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The transformer is the most critical algorithm innovation of the Nature Language Processing (NLP) field in recent years. Unlike the Recurrent Neural Network (RNN) models, Transformers can process on dimensions of sequence lengths in parallel, therefore leading to better accuracy on long sequences. However, efficient deployments of them for online services in data centers equipped with GPUs are not easy. First, more computation introduced by transformer structures makes it more challenging to meet the latency and throughput constraints of serving. Second, NLP tasks take in sentences of variable length. The variability of input dimensions brings a severe problem to efficient memory management and serving optimization. This paper designed a transformer serving system called TurboTransformers, which consists of a computing runtime and a serving framework to solve the above challenges. Three innovative features make it stand out from other similar works. An efficient parallel algorithm is proposed for GPU-based batch reduction operations, like Softmax and LayerNorm, major hot spots besides BLAS routines. A memory allocation algorithm, which better balances the memory footprint and allocation/free efficiency, is designed for variable-length input situations. A serving framework equipped with a new batch scheduler using dynamic programming achieves the optimal throughput on variable-length requests. The system can achieve the state-of-the-art transformer model serving performance on GPU platforms and can be seamlessly integrated into your PyTorch code with a few lines of code.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 9; 2023: 18; 2024: 22; 2025: 11; 2026: 5.

71 papers.

Sort:

By area:

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  3. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  4. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  5. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  6. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  7. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  8. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  9. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 167
  10. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 146
  11. 2023Fast Distributed Inference Serving for Large Language Modelscited by 143
  12. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
38 more
  1. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  2. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipelinecited by 86
  3. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instancescited by 83
  4. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  5. 2023Fairness in Serving Large Language Modelscited by 77
  6. 2024ReLU2 Wins: Discovering Efficient Activation Functions for Sparse LLMscited by 66
  7. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  8. 2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Servicescited by 44
  9. 2024Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameterscited by 37
  10. 2024Queue management for slo-oriented large language model servingcited by 36
  11. 2022ByteTransformer: A High-Performance Transformer Boosted for Variable-Length Inputscited by 32
  12. 2021Understanding Data Storage and Ingestion for Large-Scale Deep Recommendation Model Trainingcited by 28
  13. 2024DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compactioncited by 26
  14. 2023PIT: Optimization of Dynamic Sparse Deep Learning Models via Permutation Invariant Transformationcited by 25
  15. 2024Galaxy: A Resource-Efficient Collaborative Edge AI System for In-situ Transformer Inferencecited by 25
  16. 2023Task-Adaptive Tokenization: Enhancing Long-Form Text Generation Efficacy in Mental Health and Beyondcited by 22
  17. 2022A Length Adaptive Algorithm-Hardware Co-design of Transformer on FPGA Through Sparse Attention and Dynamic Pipeliningcited by 21
  18. 2021LightSeq2: Accelerated Training for Transformer-based Models on GPUscited by 19
  19. 2023The Synergy of Speculative Decoding and Batching in Serving Large Language Modelscited by 19
  20. 2024SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Servingcited by 18
  21. 2025JITServe: SLO-aware LLM Serving with Imprecise Request Informationcited by 17
  22. 2021PatrickStar: Parallel Training of Pre-trained Models via Chunk-based Memory Managementcited by 15
  23. 2025Seesaw: High-throughput LLM Inference via Model Re-shardingcited by 15
  24. 2021FastSeq: Make Sequence Generation Fastercited by 13
  25. 2022FastFold: Reducing AlphaFold Training Time from 11 Days to 67 Hourscited by 13
  26. 2024Slice-Level Scheduling for High Throughput and Load Balanced LLM Servingcited by 13
  27. 2022HelixFold: An Efficient Implementation of AlphaFold2 using PaddlePaddlecited by 12
  28. 2024Efficient Heterogeneous Large Language Model Decoding with Model-Attention Disaggregationcited by 12
  29. 2024The Case for Co-Designing Model Architectures with Hardwarecited by 11
  30. 2025LMDeploy Accelerates Mixed-Precision LLM Inference with TurboMindcited by 10
  31. 2025SPIN: Accelerating Large Language Model Inference with Heterogeneous Speculative Modelscited by 9
  32. 2021Optimizing Inference Performance of Transformers on CPUscited by 7
  33. 2023NNQS-Transformer: an Efficient and Scalable Neural Network Quantum States Approach for Ab initio Quantum Chemistrycited by 7
  34. 2025Mell: Memory-Efficient Large Language Model Serving via Multi-GPU KV Cache Managementcited by 7
  35. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inferencecited by 5
  36. 2024SoD2: Statically Optimizing Dynamic Deep Neural Networkcited by 4
  37. 2024Achieving Peak Performance for Large Language Models: A Systematic Reviewcited by 4
  38. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4

50 most cited of 71 All 71 (JSON)