mlsanityData to 7 Oct 2026

Retentive Network: A Successor to Transformer for Large Language Models

arXiv 2307.08621 ↗First posted 17 Jul 2023cs.CL cs.LGcited by 594

Not on the map. Why?

Cited by 67 in State-space sequence models (S4, Mamba), but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this work, we propose Retentive Network (RetNet) as a foundation architecture for large language models, simultaneously achieving training parallelism, low-cost inference, and good performance. We theoretically derive the connection between recurrence and attention. Then we propose the retention mechanism for sequence modeling, which supports three computation paradigms, i.e., parallel, recurrent, and chunkwise recurrent. Specifically, the parallel representation allows for training parallelism. The recurrent representation enables low-cost O(1) inference, which improves decoding throughput, latency, and GPU memory without sacrificing performance. The chunkwise recurrent representation facilitates efficient long-sequence modeling with linear complexity, where each chunk is encoded parallelly while recurrently summarizing the chunks. Experimental results on language modeling show that RetNet achieves favorable scaling results, parallel training, low-cost deployment, and efficient inference. The intriguing properties make RetNet a strong successor to Transformer for large language models. Code will be available at https://aka.ms/retnet.

Key facts. Cited by 594 papers in the map (data through 7 Oct 2026). Builds on 38 of its 44 references that are in the map. First posted on arXiv 17 Jul 2023.

Builds on

38 of 44 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2016Layer Normalizationcited by 5,579
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  10. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
26 more
  1. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  3. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  4. 2018Group Normalizationcited by 1,514
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  6. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  7. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  8. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  9. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  10. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  11. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  12. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  13. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  14. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  15. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  16. 2021Efficient Attentions for Long Document Summarizationcited by 342
  17. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
  18. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  19. 2017Searching for Activation Functionscited by 158
  20. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
  21. 2022A Length-Extrapolatable Transformercited by 123
  22. 2022Language Models are General-Purpose Interfacescited by 96
  23. 2022Structured Prompting: Scaling In-Context Learning to 1,000 Examplescited by 81
  24. 2021SummScreen: A Dataset for Abstractive Screenplay Summarizationcited by 80
  25. 2022Foundation Transformerscited by 19
  26. 2022TorchScale: Transformers at Scalecited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 34; 2024: 171; 2025: 175; 2026: 214.

594 papers.

Sort:

By area:

  1. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  2. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  3. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  4. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  5. 2024VMamba: Visual State Space Modelcited by 955
  6. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  7. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  8. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  9. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  10. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  11. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  12. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
38 more
  1. 2024Titans: Learning to Memorize at Test Timecited by 303
  2. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  3. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  4. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  5. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  6. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 208
  7. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  8. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  9. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
  10. 2023Efficient Large Language Models: A Surveycited by 183
  11. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  12. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 175
  13. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  14. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  15. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  16. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 155
  17. 2024PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognitioncited by 141
  18. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  19. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  20. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  21. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 122
  22. 2025TTT3R: 3D Reconstruction as Test-Time Trainingcited by 122
  23. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 121
  24. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  25. 2025Test-Time Training Done Rightcited by 121
  26. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 120
  27. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 118
  28. 2024DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Executioncited by 116
  29. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 111
  30. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  31. 2024Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understandingcited by 106
  32. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Taskscited by 104
  33. 2024RoboMamba: Efficient Vision-Language-Action Model for Robotic Reasoning and Manipulationcited by 102
  34. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  35. 2024Vision-RWKV: Efficient and Scalable Visual Perception with RWKV-Like Architecturescited by 97
  36. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 96
  37. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 95
  38. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89

50 most cited of 594 All 594 (JSON)