# DizzyRNN: Reparameterizing Recurrent Neural Networks for Norm-Preserving Backpropagation

arXiv: https://arxiv.org/abs/1612.04035 · HTML: https://mlsanity.com/paper/1612.04035/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Dec 2016.

## Built on by (most cited first, 5 of 5)

- [DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products](https://mlsanity.com/paper/2502.10297/) (2025)
- [Learning Connectivity with Graph Convolutional Networks for Skeleton-based Action Recognition](https://mlsanity.com/paper/2112.03328/) (2021)
- [Entangled Residual Mappings](https://mlsanity.com/paper/2206.01261/) (2022)
- [Towards Better Orthogonality Regularization with Disentangled Norm in Training Deep CNNs](https://mlsanity.com/paper/2306.09939/) (2023)
- [Convergence Analysis for Learning Orthonormal Deep Linear Neural Networks](https://mlsanity.com/paper/2311.14658/) (2023)

## Abstract (arXiv)

The vanishing and exploding gradient problems are well-studied obstacles that make it difficult for recurrent neural networks to learn long-term time dependencies. We propose a reparameterization of standard recurrent neural networks to update linear transformations in a provably norm-preserving way through Givens rotations. Additionally, we use the absolute value function as an element-wise non-linearity to preserve the norm of backpropagated signals over the entire network. We show that this reparameterization reduces the number of parameters and maintains the same algorithmic complexity as a standard recurrent neural network, while outperforming standard recurrent neural networks with orthogonal initializations and Long Short-Term Memory networks on the copy problem.