mlsanityData to 14 Oct 2026

Noisy Recurrent Neural Networks

arXiv 2102.04877 ↗First posted 9 Feb 2021stat.ML cs.LG math.DScited by 42

Not on the map. Why?

Abstract (arXiv)

We provide a general framework for studying recurrent neural networks (RNNs) trained by injecting noise into hidden states. Specifically, we consider RNNs that can be viewed as discretizations of stochastic differential equations driven by input data. This framework allows us to study the implicit regularization effect of general noise injection schemes by deriving an approximate explicit regularizer in the small noise regime. We find that, under reasonable assumptions, this implicit regularization promotes flatter minima; it biases towards models with more stable dynamics; and, in classification tasks, it favors models with larger classification margin. Sufficient conditions for global stability are obtained, highlighting the phenomenon of stochastic stabilization, where noise injection can improve stability during training. Our theory is supported by empirical results which demonstrate that the RNNs have improved robustness with respect to various input perturbations.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Builds on 58 of its 98 references that are in the map. First posted on arXiv 9 Feb 2021.

Builds on

58 of 98 references.

References · 58

  1. 2013Intriguing properties of neural networkscited by 5,155
  2. 2018Neural Ordinary Differential Equationscited by 3,779
  3. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  4. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  5. 2016Deep Networks with Stochastic Depthcited by 979
  6. 2019Hamiltonian Neural Networkscited by 712
  7. 2020Neural Controlled Differential Equations for Irregular Time Seriescited by 506
  8. 2017Stable Architectures for Deep Neural Networkscited by 341
  9. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  10. 2019Deep Lagrangian Networks: Using Physics as Model Prior for Deep Learningcited by 212
  11. 2017Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equationscited by 203
  12. 2014Recurrent Neural Network Regularizationcited by 189
38 more
  1. 2019Symplectic ODE-Net: Learning Hamiltonian Dynamics with Controlcited by 184
  2. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 184
  3. 2019Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limitcited by 176
  4. 2018Disentangling Adversarial Robustness and Generalizationcited by 172
  5. 2017Learning Koopman Invariant Subspaces for Dynamic Mode Decompositioncited by 158
  6. 2019Symplectic Recurrent Neural Networkscited by 150
  7. 2019Hamiltonian Generative Networkscited by 137
  8. 2019Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Groupcited by 134
  9. 2015A Simple Way to Initialize Recurrent Networks of Rectified Linear Unitscited by 130
  10. 2016Robust Large Margin Deep Neural Networkscited by 118
  11. 2016Sequential Neural Models with Stochastic Layerscited by 115
  12. 2020Forecasting Sequential Data using Consistent Koopman Autoencoderscited by 115
  13. 2019AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networkscited by 112
  14. 2020Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don'tcited by 95
  15. 2020Lipschitz Recurrent Neural Networkscited by 89
  16. 2018Hessian-based Analysis of Large Batch Training and Robustness to Adversariescited by 87
  17. 2020Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependenciescited by 82
  18. 2019Data-dependent Sample Complexity of Deep Neural Networks via Lipschitz Augmentationcited by 71
  19. 2020The Implicit and Explicit Regularization Effects of Dropoutcited by 71
  20. 2019Physics-informed Autoencoders for Lyapunov-stable Fluid Flow Predictioncited by 67
  21. 2019Learning Compositional Koopman Operators for Model-Based Controlcited by 67
  22. 2018Stable Recurrent Modelscited by 66
  23. 2020The Implicit Regularization of Stochastic Gradient Flow for Least Squarescited by 59
  24. 2017Regularizing Deep Neural Networks by Noise: Its Interpretation and Optimizationcited by 56
  25. 2019Physics-Informed Probabilistic Learning of Linear Embeddings of Non-linear Dynamics With Guaranteed Stabilitycited by 53
  26. 2019Machine Learning from a Continuous Viewpointcited by 50
  27. 2020Do RNN and LSTM have Long Memory?cited by 47
  28. 2018Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networkscited by 45
  29. 2020Continuous-in-Depth Neural Networkscited by 40
  30. 2020Lyapunov spectra of chaotic recurrent neural networkscited by 37
  31. 2020Explicit Regularisation in Gaussian Noise Injectionscited by 37
  32. 2019Exact expressions for double descent and implicit regularization via surrogate random designcited by 35
  33. 2020Optimizing Neural Networks via Koopman Operator Theorycited by 33
  34. 2017Theory of Deep Learning III: explaining the non-overfitting puzzlecited by 32
  35. 2020On Lyapunov Exponents for RNNs: Understanding Information Propagation Using Dynamical Systems Toolscited by 31
  36. 2020Dropout: Explicit Forms and Capacity Controlcited by 27
  37. 2019Deep Variational Koopman Models: Inferring Koopman Observations for Uncertainty-Aware Dynamics Modeling and Controlcited by 18
  38. 2019Disentangling Trainability and Generalization in Deep Neural Networkscited by 16

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 7; 2022: 12; 2023: 9; 2024: 5; 2025: 3; 2026: 6.

42 papers.

Sort:

By area:

  1. 2021Long Expressive Memory for Sequence Modelingcited by 41
  2. 2022Learning continuous models for continuous physicscited by 28
  3. 2021Noisy Feature Mixupcited by 27
  4. 2021The edge of chaos: quantum field theory and deep neural networkscited by 24
  5. 2023Generalization bounds for neural ordinary differential equations and deep residual networkscited by 21
  6. 2023Exploiting Noise as a Resource for Computation and Learning in Spiking Neural Networkscited by 18
  7. 2022NoisyMix: Boosting Model Robustness to Common Corruptionscited by 17
  8. 2021Stateful ODE-Nets using Basis Function Expansionscited by 14
  9. 2022Chaotic Regularization and Heavy-Tailed Limits for Deterministic Gradient Descentcited by 9
  10. 2026ThinkRouter: Efficient Reasoning via Routing Thinking between Latent and Discrete Spacescited by 8
  11. 2022Learning Low Dimensional State Spaces with Overparameterized Recurrent Neural Netscited by 7
  12. 2022Stabilizing Machine Learning Prediction of Dynamics: Noise and Noise-inspired Regularizationcited by 5
30 more
  1. 2022Gated Recurrent Neural Networks with Weighted Time-Delay Feedbackcited by 5
  2. 2025Robust Asymmetric Heterogeneous Federated Learning with Corrupted Clientscited by 5
  3. 2022Benefits of Additive Noise in Composing Classes with Bounded Capacitycited by 4
  4. 2023SDYN-GANs: Adversarial Learning Methods for Multistep Generative Models for General Order Stochastic Dynamicscited by 4
  5. 2024Towards Unraveling and Improving Generalization in World Modelscited by 4
  6. 2021A Differential Geometry Perspective on Orthogonal Recurrent Modelscited by 3
  7. 2021Social Neuro AI: Social Interaction as the "dark matter" of AIcited by 3
  8. 2022Lyapunov-Guided Representation of Recurrent Neural Network Performancecited by 3
  9. 2023A Distance Correlation-Based Approach to Characterize the Effectiveness of Recurrent Neural Networks for Time Series Forecastingcited by 3
  10. 2023Contaminated Multivariate Time-Series Anomaly Detection with Spatio-Temporal Graph Conditional Diffusion Modelscited by 3
  11. 2024Stratify: Unifying Multi-Step Forecasting Strategiescited by 3
  12. 2026One Step Forward and K Steps Back: Better Reasoning with Denoising Recursion Modelscited by 3
  13. 2022Learning in Feedback-driven Recurrent Spiking Neural Networks using full-FORCE Trainingcited by 2
  14. 2022p-Adic Statistical Field Theory and Deep Belief Networkscited by 2
  15. 2024Towards Robust Transcription: Exploring Noise Injection Strategies for Training Data Augmentationcited by 2
  16. 2025Ego-centric Learning of Communicative World Models for Autonomous Drivingcited by 2
  17. 2021Cluster-and-Conquer: A Framework For Time-Series Forecastingcited by 1
  18. 2022Stochastic Perturbations of Tabular Features for Non-Deterministic Inference with Automungecited by 1
  19. 2022Neural Generalized Ordinary Differential Equations with Layer-varying Parameterscited by 1
  20. 2023Dynamic Analysis and an Eigen Initializer for Recurrent Neural Networkscited by 1
  21. 2024MeMo: Meaningful, Modular Controllers via Noise Injectioncited by 1
  22. 2023On the Role of Noise in the Sample Complexity of Learning Recurrent Neural Networks: Exponential Gaps for Long Sequencescited by 0
  23. 2023Learning Stochastic Dynamical Systems as an Implicit Regularization with Graph Neural Networkscited by 0
  24. 2023The Benefit of Noise-Injection for Dynamic Gray-Box Model Creationcited by 0
  25. 2024SGD Jittering: A Training Strategy for Robust and Accurate Model-Based Architecturescited by 0
  26. 2025Variational Adaptive Noise and Dropout towards Stable Recurrent Neural Networkscited by 0
  27. 2026Paradoxical noise preference in RNNscited by 0
  28. 2026Distinguishing indistinguishable attractors: Unsupervised anomaly detection with reservoir computerscited by 0
  29. 2026NeuroRule: Making Black-Box Neural Networks Explainable through Rule-set Evolutioncited by 0
  30. 2026Where Does Randomness Matter in Neural Cellular Automata?cited by 0