mlsanityData to 7 Oct 2026

A Survey of Retentive Network

arXiv 2506.06708 ↗First posted 7 Jun 2025cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Retentive Network (RetNet) represents a significant advancement in neural network architecture, offering an efficient alternative to the Transformer. While Transformers rely on self-attention to model dependencies, they suffer from high memory costs and limited scalability when handling long sequences due to their quadratic complexity. To mitigate these limitations, RetNet introduces a retention mechanism that unifies the inductive bias of recurrence with the global dependency modeling of attention. This mechanism enables linear-time inference, facilitates efficient modeling of extended contexts, and remains compatible with fully parallelizable training pipelines. RetNet has garnered significant research interest due to its consistently demonstrated cross-domain effectiveness, achieving robust performance across machine learning paradigms including natural language processing, speech recognition, and time-series analysis. However, a comprehensive review of RetNet is still missing from the current literature. This paper aims to fill that gap by offering the first detailed survey of the RetNet architecture, its key innovations, and its diverse applications. We also explore the main challenges associated with RetNet and propose future research directions to support its continued advancement in both academic research and practical deployment.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 36 of its 121 references that are in the map. First posted on arXiv 7 Jun 2025.

Builds on

36 of 121 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016Layer Normalizationcited by 5,579
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  5. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  8. 2018Group Normalizationcited by 1,514
  9. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  10. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  11. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  12. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
24 more
  1. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  2. 2017Searching for Activation Functionscited by 158
  3. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  4. 2017Recent Advances in Recurrent Neural Networkscited by 89
  5. 2023RMT: Retentive Networks Meet Vision Transformerscited by 68
  6. 2024Improving Token-Based World Models with Parallel Observation Predictioncited by 11
  7. 2024Sable: a Performant, Efficient and Scalable Sequence Model for MARLcited by 10
  8. 2025TransRAD: Retentive Vision Transformer for Enhanced Radar Object Detectioncited by 8
  9. 2024EEGDiR: Electroencephalogram denoising network for temporal information storage and global modeling through Retentive Networkcited by 4
  10. 2024Retentive Neural Quantum States: Efficient Ans"atze for Ab Initio Quantum Chemistrycited by 4
  11. 2025Linear Attention for Efficient Bidirectional Sequence Modelingcited by 4
  12. 2024ST-RetNet: A Long-term Spatial-Temporal Traffic Flow Prediction Methodcited by 3
  13. 2023ViR: Towards Efficient Vision Retention Backbonescited by 2
  14. 2023Cross-Axis Transformer with 3D Rotary Positional Embeddingscited by 2
  15. 2023FlashVideo: A Framework for Swift Inference in Text-to-Video Generationcited by 2
  16. 2024A Contact Model based on Denoising Diffusion to Learn Variable Impedance Control for Contact-rich Manipulationcited by 2
  17. 2024Shifted Window Fourier Transform And Retention For Image Captioningcited by 2
  18. 2024Graph Retention Networks for Dynamic Graphscited by 2
  19. 2023RetSeg: Retention-based Colorectal Polyps Segmentation Networkcited by 1
  20. 2024Short-Term Photovoltaic Forecasting Model for Qualifying Uncertainty during Hazy Weathercited by 1
  21. 2024RetCompletion:High-Speed Inference Image Completion with Retentive Networkcited by 1
  22. 2024LS-EEND: Long-Form Streaming End-to-End Neural Diarization with Online Attractor Extractioncited by 1
  23. 2025SegRet: An Efficient Design for Semantic Segmentation with Retentive Networkcited by 1
  24. 2025TT-DF: A Large-Scale Diffusion-Based Dataset and Benchmark for Human Body Forgery Detectioncited by 1

Built on by

Not cited yet.