mlsanityData to 14 Oct 2026

Feedback Gradient Descent: Efficient and Stable Optimization with Orthogonality for DNNs

arXiv 2205.08385 ↗First posted 12 May 2022cs.LG cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

The optimization with orthogonality has been shown useful in training deep neural networks (DNNs). To impose orthogonality on DNNs, both computational efficiency and stability are important. However, existing methods utilizing Riemannian optimization or hard constraints can only ensure stability while those using soft constraints can only improve efficiency. In this paper, we propose a novel method, named Feedback Gradient Descent (FGD), to our knowledge, the first work showing high efficiency and stability simultaneously. FGD induces orthogonality based on the simple yet indispensable Euler discretization of a continuous-time dynamical system on the tangent bundle of the Stiefel manifold. In particular, inspired by a numerical integration method on manifolds called Feedback Integrators, we propose to instantiate it on the tangent bundle of the Stiefel manifold for the first time. In the extensive image classification experiments, FGD comprehensively outperforms the existing state-of-the-art methods in terms of accuracy, efficiency, and stability.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 48 of its 75 references that are in the map. First posted on arXiv 12 May 2022.

Builds on

48 of 75 references.

References · 48

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  5. 2016Wide Residual Networkscited by 3,070
  6. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  7. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  8. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  9. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  10. 2015Unitary Evolution Recurrent Neural Networkscited by 294
  11. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  12. 2019Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Groupcited by 134
36 more
  1. 2015A Simple Way to Initialize Recurrent Networks of Rectified Linear Unitscited by 130
  2. 2015All you need is a good initcited by 124
  3. 2019Orthogonal Convolutional Neural Networkscited by 123
  4. 2016Neural Photo Editing with Introspective Adversarial Networkscited by 109
  5. 2021Orthogonalizing Convolutional Layers with the Cayley Transformcited by 105
  6. 2016Full-Capacity Unitary Recurrent Neural Networkscited by 94
  7. 2017Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networkscited by 92
  8. 2017On orthogonality and learning recurrent networks with long term dependenciescited by 83
  9. 2017Orthogonal Recurrent Neural Networks with Scaled Cayley Transformcited by 83
  10. 2020Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transformcited by 80
  11. 2019Trivializations for Gradient-Based Optimization on Manifoldscited by 79
  12. 2016Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNscited by 65
  13. 2018Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?cited by 62
  14. 2017All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulationcited by 58
  15. 2016Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflectionscited by 57
  16. 2018On the insufficiency of existing momentum schemes for Stochastic Optimizationcited by 49
  17. 2015Natural Neural Networkscited by 44
  18. 2020Orthogonal Over-Parameterized Trainingcited by 44
  19. 2016Regularizing CNNs with Locally Constrained Decorrelationscited by 41
  20. 2020Deep Isometric Learning for Visual Recognitioncited by 39
  21. 2020Controllable Orthogonalization in Training DNNscited by 30
  22. 2014Optimization Techniques on Riemannian Manifoldscited by 27
  23. 2016Improving training of deep neural networks via Singular Value Boundingcited by 26
  24. 2016Generalized BackPropagation, 'Etude De Cas: Orthogonalitycited by 17
  25. 2016Learning Unitary Operators with Help From u(n)cited by 15
  26. 2021Convolutional Normalization: Improving Deep Convolutional Network Robustness and Trainingcited by 15
  27. 2013A Framework of Constraint Preserving Update Schemes for Optimization on Stiefel Manifoldcited by 13
  28. 2018Complex Unitary Recurrent Neural Networks using Scaled Cayley Transformcited by 12
  29. 2017Projection Based Weight Normalization for Deep Neural Networkscited by 8
  30. 2017Training Group Orthogonal Neural Networks with Privileged Informationcited by 6
  31. 2019Eigenvalue Normalized Recurrent Neural Networks for Short Term Memorycited by 5
  32. 2020Adaptive and Momentum Methods on Manifolds Through Trivializationscited by 4
  33. 2016Feedback Integratorscited by 1
  34. 2018On Controller Design for Systems on Manifolds in Euclidean Spacecited by 1
  35. 2019Model Predictive Tracking Control for Invariant Systems on Matrix Lie Groups via Stable Embedding into Euclidean Spacescited by 1
  36. 2020Tracking Controller Design for Satellite Attitude Under Unknown Constant Disturbance Using Stable Embeddingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 0; 2024: 0; 2025: 1; 2026: 1.

5 papers.

  1. 2022Toward Equation of Motion for Deep Neural Networks: Continuous-time Gradient Descent and Discretization Error Analysiscited by 13
  2. 2022Momentum Stiefel Optimizer, with Applications to Suitably-Orthogonal Attention, and Optimal Transportcited by 11
  3. 2025Non-Negative Stiefel Approximating Flow: Orthogonalish Matrix Optimization for Interpretable Embeddingscited by 1
  4. 2026Different Layers, Different Manifolds: Module-Wise Weight-Space Geometry in Transformer Optimizationcited by 1
  5. 2022Changes from Classical Statistics to Modern Statistics and Data Sciencecited by 0