mlsanityData to 7 Oct 2026

Implicit Bias of Linear RNNs

arXiv 2101.07833 ↗First posted 19 Jan 2021cs.LG cs.NE cs.SYcited by 11

Not on the map. Why?

Abstract (arXiv)

Contemporary wisdom based on empirical studies suggests that standard recurrent neural networks (RNNs) do not perform well on tasks requiring long-term memory. However, precise reasoning for this behavior is still unknown. This paper provides a rigorous explanation of this property in the special case of linear RNNs. Although this work is limited to linear RNNs, even these systems have traditionally been difficult to analyze due to their non-linear parameterization. Using recently-developed kernel regime analysis, our main result shows that linear RNNs learned from random initializations are functionally equivalent to a certain weighted 1D-convolutional network. Importantly, the weightings in the equivalent model cause an implicit bias to elements with smaller time lags in the convolution and hence, shorter memory. The degree of this bias depends on the variance of the transition kernel matrix at initialization and is related to the classic exploding and vanishing gradients problem. The theory is validated in both synthetic and real data experiments.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 34 of its 55 references that are in the map. First posted on arXiv 19 Jan 2021.

Builds on

34 of 55 references.

References · 34

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  3. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  4. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  5. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  6. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  7. 2017Deep Neural Networks as Gaussian Processescited by 617
  8. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  9. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  10. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  11. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  12. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
22 more
  1. 2015Unitary Evolution Recurrent Neural Networkscited by 291
  2. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  3. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
  4. 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 148
  5. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 146
  6. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 139
  7. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  8. 2018Deep Convolutional Networks as shallow Gaussian Processescited by 137
  9. 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
  10. 2017Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Modelscited by 115
  11. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  12. 2016Full-Capacity Unitary Recurrent Neural Networkscited by 90
  13. 2020The Recurrent Neural Tangent Kernelcited by 67
  14. 2016Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNscited by 62
  15. 2017SGD Learns the Conjugate Kernel Class of the Networkcited by 54
  16. 2019A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamicscited by 49
  17. 2018Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networkscited by 44
  18. 2020Generalization Error of Generalized Linear Models in High Dimensionscited by 25
  19. 2020RNNPool: Efficient Non-linear Pooling for RAM Constrained Inferencecited by 21
  20. 2019Training Neural Networks as Learning Data-adaptive Kernels: Provable Representation and Approximation Benefitscited by 13
  21. 2020Asymptotic errors for convex penalized linear regression beyond Gaussian matricescited by 6
  22. 2019Input-Output Equivalence of Unitary and Contractive RNNscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 3; 2023: 3; 2024: 0; 2025: 2; 2026: 1.

11 papers.

  1. 2021Noisy Recurrent Neural Networkscited by 42
  2. 2023How connectivity structure shapes rich and lazy learning in neural circuitscited by 15
  3. 2022On the Implicit Bias of Gradient Descent for Temporal Extrapolationcited by 7
  4. 2022Learning Low Dimensional State Spaces with Overparameterized Recurrent Neural Netscited by 7
  5. 2022Implicit N-grams Induced by Recurrencecited by 4
  6. 2023PAC-Bayes Generalisation Bounds for Dynamical Systems Including Stable RNNscited by 4
  7. 2025On the Expressivity of Selective State-Space Layers: A Multivariate Polynomial Approachcited by 3
  8. 2021Learning and Generalization in RNNscited by 0
  9. 2023Unraveling Feature Extraction Mechanisms in Neural Networkscited by 0
  10. 2025Flexible inference for animal learning rules using neural networkscited by 0
  11. 2026Noise Stability of Transformer Modelscited by 0