mlsanityData to 14 Oct 2026

Coordinate descent on the orthogonal group for recurrent neural network training

arXiv 2108.00051 ↗First posted 30 Jul 2021cs.LG math.OCcited by 5

Not on the map. Why?

Abstract (arXiv)

We propose to use stochastic Riemannian coordinate descent on the orthogonal group for recurrent neural network training. The algorithm rotates successively two columns of the recurrent matrix, an operation that can be efficiently implemented as a multiplication by a Givens matrix. In the case when the coordinate is selected uniformly at random at each iteration, we prove the convergence of the proposed algorithm under standard assumptions on the loss function, stepsize and minibatch noise. In addition, we numerically demonstrate that the Riemannian gradient in recurrent neural network training has an approximately sparse structure. Leveraging this observation, we propose a faster variant of the proposed algorithm that relies on the Gauss-Southwell rule. Experiments on a benchmark recurrent neural network training problem are presented to demonstrate the effectiveness of the proposed algorithm.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 32 of its 46 references that are in the map. First posted on arXiv 30 Jul 2021.

Builds on

32 of 46 references.

References · 32

  1. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  2. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  3. 2015Unitary Evolution Recurrent Neural Networkscited by 294
  4. 2015Coordinate Descent Algorithmscited by 219
  5. 2018Riemannian Adaptive Optimization Methodscited by 194
  6. 2017Spectral Norm Regularization for Improving the Generalizability of Deep Learningcited by 182
  7. 2019Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Groupcited by 134
  8. 2017Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practicecited by 132
  9. 2016Full-Capacity Unitary Recurrent Neural Networkscited by 94
  10. 2017Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networkscited by 92
  11. 2017Orthogonal Recurrent Neural Networks with Scaled Cayley Transformcited by 83
  12. 2019Trivializations for Gradient-Based Optimization on Manifoldscited by 79
20 more
  1. 2020Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networkscited by 73
  2. 2019Orthogonal Deep Neural Networkscited by 72
  3. 2016Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNscited by 65
  4. 2016Global rates of convergence for nonconvex optimization on manifoldscited by 62
  5. 2018Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?cited by 62
  6. 2016Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflectionscited by 57
  7. 2016Recurrent Orthogonal Networks and Long-Memory Taskscited by 38
  8. 2018Global Convergence of Block Coordinate Descent in Deep Learningcited by 38
  9. 2020Controllable Orthogonalization in Training DNNscited by 30
  10. 2016Improving training of deep neural networks via Singular Value Boundingcited by 26
  11. 2018McTorch, a manifold optimization library for deep learningcited by 16
  12. 2016Learning Unitary Operators with Help From u(n)cited by 15
  13. 2016Optimization on Submanifolds of Convolution Kernels in CNNscited by 15
  14. 2021Activation function design for deep networks: linearity and effective initialisationcited by 13
  15. 2018Complex Unitary Recurrent Neural Networks using Scaled Cayley Transformcited by 12
  16. 2019Coordinate Descent Without Coordinates: Tangent Subspace Descent on Riemannian Manifoldscited by 8
  17. 2013Efficient random coordinate descent algorithms for large-scale structured nonconvex optimizationcited by 5
  18. 2014Randomized Block Coordinate Descent for Online and Stochastic Optimizationcited by 5
  19. 2020Adaptive and Momentum Methods on Manifolds Through Trivializationscited by 4
  20. 2020Block Layer Decomposition schemes for training Deep Neural Networkscited by 2

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 2; 2025: 1.

5 papers.

  1. 2024BlockLLM: Memory-Efficient Adaptation of LLMs by Selecting and Optimizing the Right Coordinate Blockscited by 9
  2. 2025Times2D: Multi-Period Decomposition and Derivative Mapping for General Time Series Forecastingcited by 6
  3. 2024Riemannian coordinate descent algorithms on matrix manifoldscited by 4
  4. 2023A Block Coordinate Descent Method for Nonsmooth Composite Optimization under Orthogonality Constraintscited by 2
  5. 2023Dynamic Analysis and an Eigen Initializer for Recurrent Neural Networkscited by 1