mlsanityData to 7 Oct 2026

Efficiently Scaling Transformer Inference

arXiv 2211.05102 ↗First posted 9 Nov 2022cs.LG cs.CLcited by 529

Not on the map. Why?

Cited by 179 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We study the problem of efficient generative inference for Transformer models, in one of its most challenging settings: large deep models, with tight latency targets and long sequence lengths. Better understanding of the engineering tradeoffs for inference for large Transformer-based models is important as use cases of these models are growing rapidly throughout application areas. We develop a simple analytical model for inference efficiency to select the best multi-dimensional partitioning techniques optimized for TPU v4 slices based on the application requirements. We combine these with a suite of low-level optimizations to achieve a new Pareto frontier on the latency and model FLOPS utilization (MFU) tradeoffs on 500B+ parameter models that outperforms the FasterTransformer suite of benchmarks. We further show that with appropriate partitioning, the lower memory requirements of multiquery attention (i.e. multiple query heads share single key/value head) enables scaling up to 32x larger context lengths. Finally, we achieve a low-batch-size latency of 29ms per token during generation (using int8 weight quantization) and a 76% MFU during large-batch-size processing of input tokens, while supporting a long 2048-token context length on the PaLM 540B parameter model.

Key facts. Cited by 529 papers in the map (data through 7 Oct 2026). Builds on 37 of its 52 references that are in the map. First posted on arXiv 9 Nov 2022.

Builds on

37 of 52 references.

References · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  7. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  9. 2020Reformer: The Efficient Transformercited by 1,658
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  11. 2020Rethinking Attention with Performerscited by 1,584
  12. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
25 more
  1. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  2. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  3. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  4. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  5. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  6. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  7. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  8. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  9. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  10. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  11. 2022Confident Adaptive Language Modelingcited by 297
  12. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  13. 2018LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networkscited by 272
  14. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  15. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 177
  16. 2019Adaptive Attention Span in Transformerscited by 148
  17. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  18. 2020HAT: Hardware-Aware Transformers for Efficient Natural Language Processingcited by 136
  19. 2021GSPMD: General and Scalable Parallelization for ML Computation Graphscited by 109
  20. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inferencecited by 96
  21. 2021Sparse is Enough in Scaling Transformerscited by 86
  22. 2020Compression of Deep Learning Models for Text: A Surveycited by 64
  23. 2020Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruningcited by 39
  24. 2021Pareto-Optimal Quantized ResNet Is Mostly 4-bitcited by 26
  25. 2020Successfully Applying the Stabilized Lottery Ticket Hypothesis to the Transformer Architecturecited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 65; 2024: 122; 2025: 146; 2026: 193.

529 papers.

Sort:

By area:

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  6. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  7. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  8. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  9. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  10. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  11. 2024Yi: Open Foundation Models by 01.AIcited by 706
  12. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
38 more
  1. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  2. 2023The Falcon Series of Open Language Modelscited by 526
  3. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  4. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  5. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
  6. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
  7. 2023Challenges and Applications of Large Language Modelscited by 251
  8. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
  9. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  10. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  11. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  12. 2024Many-Shot In-Context Learningcited by 203
  13. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  14. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
  15. 2023Efficient Large Language Models: A Surveycited by 183
  16. 2023ChipNeMo: Domain-Adapted LLMs for Chip Designcited by 168
  17. 2025Sundial: A Family of Highly Capable Time Series Foundation Modelscited by 168
  18. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 168
  19. 2023Speculative Decoding with Big Little Decodercited by 163
  20. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillationcited by 160
  21. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  22. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  23. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 146
  24. 2025MUSt3R: Multi-view Network for Stereo 3D Reconstructioncited by 139
  25. 2024Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inferencecited by 137
  26. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  27. 2024A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthinesscited by 125
  28. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 124
  29. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 122
  30. 2023Teaching Arithmetic to Small Transformerscited by 121
  31. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  32. 2023SparQ Attention: Bandwidth-Efficient LLM Inferencecited by 119
  33. 2023Baldur: Whole-Proof Generation and Repair with Large Language Modelscited by 118
  34. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  35. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 112
  36. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decodingcited by 103
  37. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 98
  38. 2025Cameras as Relative Positional Encodingcited by 94

50 most cited of 529 All 529 (JSON)