mlsanityData to 7 Oct 2026

Orthogonalising gradients to speed up neural network optimisation

arXiv 2202.07052 ↗First posted 14 Feb 2022cs.LGcited by 25

Not on the map. Why?

Abstract (arXiv)

The optimisation of neural networks can be sped up by orthogonalising the gradients before the optimisation step, ensuring the diversification of the learned representations. We orthogonalise the gradients of the layer's components/filters with respect to each other to separate out the intermediate representations. Our method of orthogonalisation allows the weights to be used more flexibly, in contrast to restricting the weights to an orthogonalised sub-space. We tested this method on ImageNet and CIFAR-10 resulting in a large decrease in learning time, and also obtain a speed-up on the semi-supervised learning BarlowTwins. We obtain similar accuracy to SGD without fine-tuning and better accuracy for na"ively chosen hyper-parameters.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 11 of its 18 references that are in the map. First posted on arXiv 14 Feb 2022.

Builds on

11 of 18 references.

References · 11

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  5. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  6. 2017Large Batch Training of Convolutional Networkscited by 529
  7. 2019Orthogonal Convolutional Neural Networkscited by 122
  8. 2019Orthogonal Deep Neural Networkscited by 72
  9. 2018Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?cited by 62
  10. 2017All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulationcited by 58
  11. 2016Improving training of deep neural networks via Singular Value Boundingcited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 12; 2026: 12.

25 papers.

Sort:

By area:

  1. 2025Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimizationcited by 81
  2. 2025Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal?cited by 30
  3. 2026Convergence of Muon with Newton-Schulzcited by 30
  4. 2026Preconditioning Benefits of Spectral Orthogonalization in Muoncited by 28
  5. 2025Beyond the Ideal: Analyzing the Inexact Muon Updatecited by 23
  6. 2024SWAN: SGD with Normalization and Whitening Enables Stateless LLM Trainingcited by 20
  7. 2025LiMuon: Light and Fast Muon Optimizer for Large Modelscited by 20
  8. 2025Generalized Gradient Norm Clipping & Non-Euclidean (L0,L1)-Smoothnesscited by 17
  9. 2025Low-rank Orthogonalization for Large-scale Matrix Optimization with Applications to Foundation Model Trainingcited by 17
  10. 2025SUMO: Subspace-Aware Moment-Orthogonalization for Accelerating Memory-Efficient LLM Trainingcited by 16
  11. 2025What Really Matters in Matrix-Whitening Optimizers?cited by 15
  12. 2026Insights on Muon from Simple Quadraticscited by 11
13 more
  1. 2026On the Width Scaling of Neural Optimizers Under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfercited by 11
  2. 2025Optimizing Rank for High-Fidelity Implicit Neural Representationscited by 7
  3. 2025SNOO: Step-K Nesterov Outer Optimizer - The Surprising Effectiveness of Nesterov Momentum Applied to Pseudo-Gradientscited by 6
  4. 2026Muown: Row-Norm Control for Muon Optimizationcited by 6
  5. 2026Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routerscited by 5
  6. 2026Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformationcited by 2
  7. 2026Ringmaster LMO: Asynchronous Linear Minimization Oracle Momentum Methodcited by 2
  8. 2026The Spectral Dynamics and Noise Geometry of Muoncited by 2
  9. 2025OrthoGrad Improves Neural Calibrationcited by 1
  10. 2026Celo2: Towards Learned Optimization Free Lunchcited by 1
  11. 2026Free Heavy-Tailed Lunch for Muon: A Theoretical Justification of Empirical Successcited by 1
  12. 2025AuON: A Linear-time Alternative to Orthogonal Momentum Updatescited by 0
  13. 2026Reassessing Muon for Matrix Factorizationcited by 0