# A Survey of Retentive Network

arXiv: https://arxiv.org/abs/2506.06708 · HTML: https://mlsanity.com/paper/2506.06708/

## Key facts

- No citing papers in the map yet (data through 7 Oct 2026).
- Builds on 36 of its 121 references that are in the map.
- First posted on arXiv 7 Jun 2025.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Gated Linear Attention Transformers with Hardware-Efficient Training](https://mlsanity.com/paper/2312.06635/) (2023)
- [Hyena Hierarchy: Towards Larger Convolutional Language Models](https://mlsanity.com/paper/2302.10866/) (2023)
- [Parallelizing Linear Transformers with the Delta Rule over Sequence Length](https://mlsanity.com/paper/2406.06484/) (2024)
- [Searching for Activation Functions](https://mlsanity.com/paper/1710.05941/) (2017)
- [You Only Cache Once: Decoder-Decoder Architectures for Language Models](https://mlsanity.com/paper/2405.05254/) (2024)
- [Recent Advances in Recurrent Neural Networks](https://mlsanity.com/paper/1801.01078/) (2017)
- [RMT: Retentive Networks Meet Vision Transformers](https://mlsanity.com/paper/2309.11523/) (2023)
- [Improving Token-Based World Models with Parallel Observation Prediction](https://mlsanity.com/paper/2402.05643/) (2024)
- [Sable: a Performant, Efficient and Scalable Sequence Model for MARL](https://mlsanity.com/paper/2410.01706/) (2024)
- [TransRAD: Retentive Vision Transformer for Enhanced Radar Object Detection](https://mlsanity.com/paper/2501.17977/) (2025)
- [EEGDiR: Electroencephalogram denoising network for temporal information storage and global modeling through Retentive Network](https://mlsanity.com/paper/2404.15289/) (2024)
- [Retentive Neural Quantum States: Efficient Ans"atze for Ab Initio Quantum Chemistry](https://mlsanity.com/paper/2411.03900/) (2024)
- [Linear Attention for Efficient Bidirectional Sequence Modeling](https://mlsanity.com/paper/2502.16249/) (2025)
- [ST-RetNet: A Long-term Spatial-Temporal Traffic Flow Prediction Method](https://mlsanity.com/paper/2407.11074/) (2024)
- [ViR: Towards Efficient Vision Retention Backbones](https://mlsanity.com/paper/2310.19731/) (2023)
- [Cross-Axis Transformer with 3D Rotary Positional Embeddings](https://mlsanity.com/paper/2311.07184/) (2023)
- [FlashVideo: A Framework for Swift Inference in Text-to-Video Generation](https://mlsanity.com/paper/2401.00869/) (2023)
- [A Contact Model based on Denoising Diffusion to Learn Variable Impedance Control for Contact-rich Manipulation](https://mlsanity.com/paper/2403.13221/) (2024)
- [Shifted Window Fourier Transform And Retention For Image Captioning](https://mlsanity.com/paper/2408.13963/) (2024)
- [Graph Retention Networks for Dynamic Graphs](https://mlsanity.com/paper/2411.11259/) (2024)

## Abstract (arXiv)

Retentive Network (RetNet) represents a significant advancement in neural network architecture, offering an efficient alternative to the Transformer. While Transformers rely on self-attention to model dependencies, they suffer from high memory costs and limited scalability when handling long sequences due to their quadratic complexity. To mitigate these limitations, RetNet introduces a retention mechanism that unifies the inductive bias of recurrence with the global dependency modeling of attention. This mechanism enables linear-time inference, facilitates efficient modeling of extended contexts, and remains compatible with fully parallelizable training pipelines. RetNet has garnered significant research interest due to its consistently demonstrated cross-domain effectiveness, achieving robust performance across machine learning paradigms including natural language processing, speech recognition, and time-series analysis. However, a comprehensive review of RetNet is still missing from the current literature. This paper aims to fill that gap by offering the first detailed survey of the RetNet architecture, its key innovations, and its diverse applications. We also explore the main challenges associated with RetNet and propose future research directions to support its continued advancement in both academic research and practical deployment.