# On the Variance of Unbiased Online Recurrent Optimization

arXiv: https://arxiv.org/abs/1902.02405 · HTML: https://mlsanity.com/paper/1902.02405/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Feb 2019.

## Built on by (most cited first, 7 of 7)

- [Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping](https://mlsanity.com/paper/2110.01765/) (2021)
- [Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategies](https://mlsanity.com/paper/2112.13835/) (2021)
- [Real-Time Recurrent Learning using Trace Units in Reinforcement Learning](https://mlsanity.com/paper/2409.01449/) (2024)
- [AGaLiTe: Approximate Gated Linear Transformers for Online Reinforcement Learning](https://mlsanity.com/paper/2310.15719/) (2023)
- [Online hyperparameter optimization by real-time recurrent learning](https://mlsanity.com/paper/2102.07813/) (2021)
- [Beyond accuracy: generalization properties of bio-plausible temporal credit assignment rules](https://mlsanity.com/paper/2206.00823/) (2022)
- [Deep-Graph-Sprints: Accelerated Representation Learning in Continuous-Time Dynamic Graphs](https://mlsanity.com/paper/2407.07712/) (2024)

## Abstract (arXiv)

The recently proposed Unbiased Online Recurrent Optimization algorithm (UORO, arXiv:1702.05043) uses an unbiased approximation of RTRL to achieve fully online gradient-based learning in RNNs. In this work we analyze the variance of the gradient estimate computed by UORO, and propose several possible changes to the method which reduce this variance both in theory and practice. We also contribute significantly to the theoretical and intuitive understanding of UORO (and its existing variance reduction technique), and demonstrate a fundamental connection between its gradient estimate and the one that would be computed by REINFORCE if small amounts of noise were added to the RNN's hidden units.