mlsanityData to 7 Oct 2026

Fast Transformer Decoding: One Write-Head is All You Need

arXiv 1911.02150 ↗First posted 6 Nov 2019cs.NE cs.CL cs.LGcited by 700

Not on the map. Why?

Cited by 272 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Multi-head attention layers, as used in the Transformer neural sequence model, are a powerful alternative to RNNs for moving information across and between sequences. While training these layers is generally fast and simple, due to parallelizability across the length of the sequence, incremental inference (where such paralleization is impossible) is often slow, due to the memory-bandwidth cost of repeatedly loading the large "keys" and "values" tensors. We propose a variant called multi-query attention, where the keys and values are shared across all of the different attention "heads", greatly reducing the size of these tensors and hence the memory bandwidth requirements of incremental decoding. We verify experimentally that the resulting models can indeed be much faster to decode, and incur only minor quality degradation from the baseline.

Key facts. Cited by 700 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 8; 2022: 10; 2023: 69; 2024: 213; 2025: 196; 2026: 203.

700 papers.

Sort:

By area:

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  4. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  5. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  7. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  9. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  10. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  11. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  12. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
38 more
  1. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  2. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  3. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  4. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  5. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  6. 2023StarCoder: may the source be with you!cited by 790
  7. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
  8. 2024Yi: Open Foundation Models by 01.AIcited by 706
  9. 2024Autoregressive Image Generation without Vector Quantizationcited by 644
  10. 2023A Comprehensive Overview of Large Language Modelscited by 639
  11. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  12. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  13. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  14. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  15. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  16. 2022Efficiently Scaling Transformer Inferencecited by 529
  17. 2023The Falcon Series of Open Language Modelscited by 526
  18. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  19. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  20. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  21. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  22. 2021A Survey of Transformerscited by 399
  23. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  24. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  25. 2025Muon is Scalable for LLM Trainingcited by 381
  26. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  27. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  28. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  29. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  30. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283
  31. 2024The Power of Noise: Redefining Retrieval for RAG Systemscited by 275
  32. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 266
  33. 2021Generating Images with Sparse Representationscited by 262
  34. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
  35. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  36. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  37. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
  38. 2026Gemma 4 Technical Reportcited by 187

50 most cited of 700 All 700 (JSON)