# Training Recurrent Neural Networks by Diffusion

arXiv: https://arxiv.org/abs/1601.04114 · HTML: https://mlsanity.com/paper/1601.04114/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jan 2016.

## Built on by (most cited first, 9 of 9)

- [ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks](https://mlsanity.com/paper/2102.11600/) (2021)
- [Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscape](https://mlsanity.com/paper/2201.08025/) (2022)
- [Sharpness-Aware Data Poisoning Attack](https://mlsanity.com/paper/2305.14851/) (2023)
- [Seeking Consistent Flat Minima for Better Domain Generalization via Refining Loss Landscapes](https://mlsanity.com/paper/2412.13573/) (2024)
- [Entropy-MCMC: Sampling from Flat Basins with Ease](https://mlsanity.com/paper/2310.05401/) (2023)
- [Adversarial Training Makes Weight Loss Landscape Sharper in Logistic Regression](https://mlsanity.com/paper/2102.02950/) (2021)
- [Deep Contrastive Graph Representation via Adaptive Homotopy Learning](https://mlsanity.com/paper/2106.09244/) (2021)
- [EuclidNets: An Alternative Operation for Efficient Inference of Deep Learning Models](https://mlsanity.com/paper/2212.11803/) (2022)
- [WaterMAS: Sharpness-Aware Maximization for Neural Network Watermarking](https://mlsanity.com/paper/2409.03902/) (2024)

## Abstract (arXiv)

This work presents a new algorithm for training recurrent neural networks (although ideas are applicable to feedforward networks as well). The algorithm is derived from a theory in nonconvex optimization related to the diffusion equation. The contributions made in this work are two fold. First, we show how some seemingly disconnected mechanisms used in deep learning such as smart initialization, annealed learning rate, layerwise pretraining, and noise injection (as done in dropout and SGD) arise naturally and automatically from this framework, without manually crafting them into the algorithms. Second, we present some preliminary results on comparing the proposed method against SGD. It turns out that the new algorithm can achieve similar level of generalization accuracy of SGD in much fewer number of epochs.