mlsanityData to 14 Oct 2026

Variational Adaptive Noise and Dropout towards Stable Recurrent Neural Networks

arXiv 2506.01350 ↗First posted 2 Jun 2025cs.LG cs.ROcited by 0

Not on the map. Why?

Abstract (arXiv)

This paper proposes a novel stable learning theory for recurrent neural networks (RNNs), so-called variational adaptive noise and dropout (VAND). As stabilizing factors for RNNs, noise and dropout on the internal state of RNNs have been separately confirmed in previous studies. We reinterpret the optimization problem of RNNs as variational inference, showing that noise and dropout can be derived simultaneously by transforming the explicit regularization term arising in the optimization problem into implicit regularization. Their scale and ratio can also be adjusted appropriately to optimize the main objective of RNNs, respectively. In an imitation learning scenario with a mobile manipulator, only VAND is able to imitate sequential and periodic behaviors as instructed. https://youtu.be/UOho3Xr6A2w

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 23 references that are in the map. First posted on arXiv 2 Jun 2025.

Builds on

15 of 23 references.

References · 15

  1. 2013Auto-Encoding Variational Bayescited by 4,053
  2. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  3. 2022DayDreamer: World Models for Physical Robot Learningcited by 400
  4. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 320
  5. 2015A Theoretically Grounded Application of Dropout in Recurrent Neural Networkscited by 161
  6. 2020A Spatio-temporal Transformer for 3D Human Motion Predictioncited by 133
  7. 2017Implicit Regularization in Deep Learningcited by 90
  8. 2020Lipschitz Recurrent Neural Networkscited by 89
  9. 2017On orthogonality and learning recurrent networks with long term dependenciescited by 83
  10. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 61
  11. 2021Noisy Recurrent Neural Networkscited by 42
  12. 2017Unbiasing Truncated Backpropagation Through Timecited by 37
3 more
  1. 2021Transformer-based deep imitation learning for dual-arm robot manipulationcited by 32
  2. 2023Bridging Discrete and Backpropagation: Straight-Through and Beyondcited by 31
  3. 2022AdaTerm: Adaptive T-Distribution Estimated Robust Moments for Noise-Robust Stochastic Gradient Optimizationcited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Recurrent neural networks: vanishing and exploding gradients are not the end of the story
  2. 2023Scalable and Efficient Continual Learning from Demonstration via a Hypernetwork-generated Stable Dynamics Model
  3. 2021Machine Unlearning via Algorithmic Stability
  4. 2024Control-Theoretic Techniques for Online Adaptation of Deep Neural Networks in Dynamical Systems
  5. 2023Posterior sampling algorithms for unsupervised speech enhancement with recurrent variational autoencoder
  6. 2026GAC: Noise-Aware Adaptive Mixing for Hybrid SFT-RL Post-Training
  7. 2026When Learning Hurts: Fixed-Pole RNN for Real-Time Online Training
  8. 2023Variational excess risk bound for general state space models