mlsanityData to 14 Oct 2026

Fixed-Point RNNs: Interpolating from Diagonal to Dense

arXiv 2503.10799 ↗First posted 13 Mar 2025cs.LGcited by 5

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

Linear recurrent neural networks (RNNs) and state-space models (SSMs) such as Mamba have become promising alternatives to softmax-attention as sequence mixing layers in Transformer architectures. Current models, however, do not exhibit the full state-tracking expressivity of RNNs because they rely on channel-wise (i.e. diagonal) sequence mixing. In this paper, we investigate parameterizations of a large class of dense linear RNNs as fixed-points of parallelizable diagonal linear RNNs. The resulting models can naturally trade expressivity for efficiency at a fixed number of parameters and achieve state-of-the-art results on the state-tracking benchmarks A5 and S5, while matching performance on copying and other tasks.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 53 of its 74 references that are in the map. First posted on arXiv 13 Mar 2025.

Builds on

53 of 74 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024On the Expressiveness and Length Generalization of Selective State-Space Models on Regular Languages26 Dec
  2. 2024Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues19 Nov
  3. 2024Mimetic Initialization Helps State Space Models Learn to Recall14 Oct
  4. 2024An Empirical Study of Mamba-based Language Models12 Jun
  5. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality31 May
  6. 2024Theoretical Foundations of Deep Selective State-Space Models29 Feb
  7. 2024Simple linear attention language models balance the recall-throughput tradeoff28 Feb
  8. 2023Gated Linear Attention Transformers with Hardware-Efficient Training11 Dec
  9. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec
  10. 2022Simplified State Space Layers for Sequence Modeling9 Aug
  11. 2021Efficiently Modeling Long Sequences with Structured State Spaces31 Oct

Other references · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018Neural Ordinary Differential Equationscited by 3,779
  3. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  4. 2020Rethinking Attention with Performerscited by 1,601
  5. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  6. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  7. 2018Universal Transformerscited by 647
  8. 2019Deep Equilibrium Modelscited by 615
  9. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  10. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  11. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 506
  12. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
30 more
  1. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 415
  2. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 408
  3. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
  4. 2024xLSTM: Extended Long Short-Term Memorycited by 327
  5. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 314
  6. 2015Unitary Evolution Recurrent Neural Networkscited by 294
  7. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 242
  8. 2025Reasoning with Latent Thoughts: On the Power of Looped Transformerscited by 240
  9. 2023Looped Transformers as Programmable Computerscited by 237
  10. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 216
  11. 2022The Parallelism Tradeoff: Limitations of Log-Precision Transformerscited by 193
  12. 2024The Illusion of State in State-Space Modelscited by 187
  13. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 159
  14. 2017Parallelizing Linear Recurrent Neural Nets Over Sequence Lengthcited by 127
  15. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 127
  16. 2019Implicit Deep Learningcited by 124
  17. 2021Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networkscited by 117
  18. 2017Orthogonal Recurrent Neural Networks with Scaled Cayley Transformcited by 83
  19. 2021Stabilizing Equilibrium Models by Jacobian Regularizationcited by 72
  20. 2018Reviving and Improving Recurrent Back-Propagationcited by 69
  21. 2021Skyformer: Remodel Self-Attention with Gaussian Kernel and Nystr"om Methodcited by 63
  22. 2020Learning Associative Inference Using Fast Weight Memorycited by 51
  23. 2024The Expressive Capacity of State Space Models: A Formal Language Perspectivecited by 46
  24. 2023Parallelizing non-linear sequential models over the sequence lengthcited by 45
  25. 2024Towards Scalable and Stable Parallelization of Nonlinear RNNscited by 38
  26. 2024Artificial Kuramoto Oscillatory Neuronscited by 38
  27. 2019On the Computational Power of RNNscited by 35
  28. 2024Recurrent neural networks: vanishing and exploding gradients are not the end of the storycited by 25
  29. 2025Implicit Language Models are RNNs: Balancing Parallelization and Expressivitycited by 22
  30. 2024Genus expansion for non-linear random matrix ensembles with applications to neural networkscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025Structured Sparse Transition Matrices to Enable State Tracking in State-Space Modelscited by 16
  2. 2025Fast weight programming and linear transformers: from machine learning to neurobiologycited by 8
  3. 2026SHiPPO: Recurrent Memory with Transported Polynomial Projectionscited by 1
  4. 2026Faithful Embeddings of Irregular and Asynchronous Data for Online Log-NCDEscited by 0
  5. 2026Riccati State Space Models: Non-iterative Parallelization for Nonlinear Sequence Modelingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Improved state mixing in higher-order and block diagonal linear recurrent networks
  2. 2025DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products
  3. 2022Simplifying and Understanding State Space Models with Diagonal Linear RNNs
  4. 2024Transformers are Multi-State RNNs
  5. 2026M2RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling
  6. 2026Improving the Performance and Learning Stability of Parallelizable RNNs Designed for Ultra-Low Power Applications
  7. 2024Does Transformer Interpretability Transfer to RNNs?
  8. 2026Length-Independent State Tracking Under a Parallel Scan