mlsanityData to 14 Oct 2026

Random orthogonal additive filters: a solution to the vanishing/exploding gradient of deep neural networks

arXiv 2210.01245 ↗First posted 3 Oct 2022cs.LG cs.AI cs.NAcited by 5

Not on the map. Why?

Abstract (arXiv)

Since the recognition in the early nineties of the vanishing/exploding (V/E) gradient issue plaguing the training of neural networks (NNs), significant efforts have been exerted to overcome this obstacle. However, a clear solution to the V/E issue remained elusive so far. In this manuscript a new architecture of NN is proposed, designed to mathematically prevent the V/E issue to occur. The pursuit of approximate dynamical isometry, i.e. parameter configurations where the singular values of the input-output Jacobian are tightly distributed around 1, leads to the derivation of a NN's architecture that shares common traits with the popular Residual Network model. Instead of skipping connections between layers, the idea is to filter the previous activations orthogonally and add them to the nonlinear activations of the next layer, realising a convex combination between them. Remarkably, the impossibility for the gradient updates to either vanish or explode is demonstrated with analytical bounds that hold even in the infinite depth case. The effectiveness of this method is empirically proved by means of training via backpropagation an extremely deep multilayer perceptron of 50k layers, and an Elman NN to learn long-term dependencies in the input of 10k time steps in the past. Compared with other architectures specifically devised to deal with the V/E problem, e.g. LSTMs for recurrent NNs, the proposed model is way simpler yet more effective. Surprisingly, a single layer vanilla RNN can be enhanced to reach state of the art performance, while converging super fast; for instance on the psMNIST task, it is possible to get test accuracy of over 94% in the first epoch, and over 98% after just 10 epochs.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 36 of its 75 references that are in the map. First posted on arXiv 3 Oct 2022.

Builds on

36 of 75 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  5. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,304
  6. 2016An overview of gradient descent optimization algorithmscited by 1,033
  7. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  8. 2015Unitary Evolution Recurrent Neural Networkscited by 294
  9. 2015Highway Networkscited by 260
  10. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  11. 2018Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNNcited by 143
  12. 2019Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Groupcited by 134
24 more
  1. 2017Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practicecited by 132
  2. 2015A Simple Way to Initialize Recurrent Networks of Rectified Linear Unitscited by 130
  3. 2015All you need is a good initcited by 124
  4. 2019AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networkscited by 112
  5. 2016Full-Capacity Unitary Recurrent Neural Networkscited by 94
  6. 2020Lipschitz Recurrent Neural Networkscited by 89
  7. 2017On orthogonality and learning recurrent networks with long term dependenciescited by 83
  8. 2017Orthogonal Recurrent Neural Networks with Scaled Cayley Transformcited by 83
  9. 2017Dilated Recurrent Neural Networkscited by 83
  10. 2017Mean Field Residual Networks: On the Edge of Chaoscited by 83
  11. 2020Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependenciescited by 82
  12. 2019Trivializations for Gradient-Based Optimization on Manifoldscited by 79
  13. 2016Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNscited by 65
  14. 2018Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networkscited by 45
  15. 2019FastGRNN: A Fast, Accurate, Stable and Tiny Kilobyte Sized Gated Recurrent Neural Networkcited by 38
  16. 2016Recurrent Batch Normalizationcited by 36
  17. 2019Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamicscited by 29
  18. 2021Hamiltonian Deep Neural Networks Guaranteeing Non-vanishing Gradients by Designcited by 18
  19. 2019Dynamical Isometry and a Mean Field Theory of LSTMs and GRUscited by 17
  20. 2019Towards Non-saturating Recurrent Units for Modelling Long-term Dependenciescited by 15
  21. 2019Deep Independently Recurrent Neural Network (IndRNN)cited by 14
  22. 2021Activated Gradients for Deep Neural Networkscited by 14
  23. 2020Shuffling Recurrent Neural Networkscited by 11
  24. 2021Least Redundant Gated Recurrent Neural Networkcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2026CAWI: Copula-Aligned Weight Initialization for Randomized Neural Networkscited by 3
  2. 2025Time-Scale Coupling Between States and Parameters in Recurrent Neural Networkscited by 2
  3. 2025Analytic Personalized Federated Meta-Learningcited by 0
  4. 2025Deep Residual Echo State Networks: exploring residual orthogonal connections in untrained Recurrent Neural Networkscited by 0
  5. 2025Multimodal Deep Learning for Prediction of Progression-Free Survival in Patients with Neuroendocrine Tumors Undergoing 177Lu-based Peptide Receptor Radionuclide Therapycited by 0