# Stabilizing Gradients for Deep Neural Networks via Efficient SVD Parameterization

arXiv: https://arxiv.org/abs/1803.09327 · HTML: https://mlsanity.com/paper/1803.09327/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Mar 2018.

## Built on by (most cited first, 30 of 35)

- [Parallelizing Linear Transformers with the Delta Rule over Sequence Length](https://mlsanity.com/paper/2406.06484/) (2024)
- [Deep networks for system identification: a Survey](https://mlsanity.com/paper/2301.12832/) (2023)
- [Physics-Informed Machine Learning for Modeling and Control of Dynamical Systems](https://mlsanity.com/paper/2306.13867/) (2023)
- [Framing RNN as a kernel method: A neural ODE approach](https://mlsanity.com/paper/2106.01202/) (2021)
- [Principled Multimodal Representation Learning](https://mlsanity.com/paper/2507.17343/) (2025)
- [Hamiltonian Deep Neural Networks Guaranteeing Non-vanishing Gradients by Design](https://mlsanity.com/paper/2105.13205/) (2021)
- [From Generalization Analysis to Optimization Designs for State Space Models](https://mlsanity.com/paper/2405.02670/) (2024)
- [Constrained Block Nonlinear Neural Dynamical Models](https://mlsanity.com/paper/2101.01864/) (2021)
- [Structured Dropout Variational Inference for Bayesian Neural Networks](https://mlsanity.com/paper/2102.07927/) (2021)
- [Scaling-up Diverse Orthogonal Convolutional Networks with a Paraunitary Framework](https://mlsanity.com/paper/2106.09121/) (2021)
- [Spectral Tensor Train Parameterization of Deep Learning Layers](https://mlsanity.com/paper/2103.04217/) (2021)
- [Recognition of handwritten MNIST digits on low-memory 2 Kb RAM Arduino board using LogNNet reservoir neural network](https://mlsanity.com/paper/2105.02953/) (2021)
- [On the Provable Generalization of Recurrent Neural Networks](https://mlsanity.com/paper/2109.14142/) (2021)
- [Stable Long-Term Recurrent Video Super-Resolution](https://mlsanity.com/paper/2112.08950/) (2021)
- [Neural Gaussian Scale-Space Fields](https://mlsanity.com/paper/2405.20980/) (2024)
- [Householder Projector for Unsupervised Latent Semantics Discovery](https://mlsanity.com/paper/2307.08012/) (2023)
- [Generalization Error Analysis for Selective State-Space Models Through the Lens of Attention](https://mlsanity.com/paper/2502.01473/) (2025)
- [On the Stochastic Stability of Deep Markov Models](https://mlsanity.com/paper/2111.04601/) (2021)
- [Streaming, fast and accurate on-device Inverse Text Normalization for Automatic Speech Recognition](https://mlsanity.com/paper/2211.03721/) (2022)
- [PAC-Bayesian bounds for learning LTI-ss systems with input from empirical loss](https://mlsanity.com/paper/2303.16816/) (2023)
- [A Generalization Bound for a Family of Implicit Networks](https://mlsanity.com/paper/2410.07427/) (2024)
- [A Method for Medical Data Analysis Using the LogNNet for Clinical Decision Support Systems and Edge Computing in Healthcare](https://mlsanity.com/paper/2108.02428/) (2021)
- [State space models can express n-gram languages](https://mlsanity.com/paper/2306.17184/) (2023)
- [Length independent generalization bounds for deep SSM architectures via Rademacher contraction and stability constraints](https://mlsanity.com/paper/2405.20278/) (2024)
- [Generalization and Risk Bounds for Recurrent Neural Networks](https://mlsanity.com/paper/2411.02784/) (2024)
- [Revisiting Glorot Initialization for Long-Range Linear Recurrences](https://mlsanity.com/paper/2505.19827/) (2025)
- [PAC-Bayesian-Like Error Bound for a Class of Linear Time-Invariant Stochastic State-Space Models](https://mlsanity.com/paper/2212.14838/) (2022)
- [Quantized Approximately Orthogonal Recurrent Neural Networks](https://mlsanity.com/paper/2402.04012/) (2024)
- [SVMax: A Feature Embedding Regularizer](https://mlsanity.com/paper/2103.02770/) (2021)
- [A Lightweight and Gradient-Stable Neural Layer](https://mlsanity.com/paper/2106.04088/) (2021)

## Abstract (arXiv)

Vanishing and exploding gradients are two of the main obstacles in training deep neural networks, especially in capturing long range dependencies in recurrent neural networks~(RNNs). In this paper, we present an efficient parametrization of the transition matrix of an RNN that allows us to stabilize the gradients that arise in its training. Specifically, we parameterize the transition matrix by its singular value decomposition(SVD), which allows us to explicitly track and control its singular values. We attain efficiency by using tools that are common in numerical linear algebra, namely Householder reflectors for representing the orthogonal matrices that arise in the SVD. By explicitly controlling the singular values, our proposed Spectral-RNN method allows us to easily solve the exploding gradient problem and we observe that it empirically solves the vanishing gradient issue to a large extent. We note that the SVD parameterization can be used for any rectangular weight matrix, hence it can be easily extended to any deep neural network, such as a multi-layer perceptron. Theoretically, we demonstrate that our parameterization does not lose any expressive power, and show how it controls generalization of RNN for the classification task. %, and show how it potentially makes the optimization process easier. Our extensive experimental results also demonstrate that the proposed framework converges faster, and has good generalization, especially in capturing long range dependencies, as shown on the synthetic addition and copy tasks, as well as on MNIST and Penn Tree Bank data sets.