mlsanityData to 7 Oct 2026

Learning by Turning: Neural Architecture Aware Optimisation

arXiv 2102.07227 ↗First posted 14 Feb 2021cs.NE cs.LGcited by 23

Not on the map. Why?

Abstract (arXiv)

Descent methods for deep networks are notoriously capricious: they require careful tuning of step size, momentum and weight decay, and which method will work best on a new benchmark is a priori unclear. To address this problem, this paper conducts a combined study of neural architecture and optimisation, leading to a new optimiser called Nero: the neuronal rotator. Nero trains reliably without momentum or weight decay, works in situations where Adam and SGD fail, and requires little to no learning rate tuning. Also, Nero's memory footprint is ~ square root that of Adam or LAMB. Nero combines two ideas: (1) projected gradient descent over the space of balanced networks; (2) neuron-specific updates, where the step size sets the angle through which each neuron's hyperplane turns. The paper concludes by discussing how this geometric connection between architecture and optimisation may impact theories of generalisation in deep learning.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 29 of its 52 references that are in the map. First posted on arXiv 14 Feb 2021.

Builds on

29 of 52 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  7. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  8. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  9. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  10. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  11. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  12. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
17 more
  1. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  2. 2019On the Convergence of Adam and Beyondcited by 834
  3. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  4. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  5. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 248
  6. 2017High-dimensional dynamics of generalization error in neural networkscited by 238
  7. 2018Deep learning generalizes because the parameter-function map is biased towards simple functionscited by 180
  8. 2021Characterizing signal propagation to close the performance gap in unnormalized ResNetscited by 87
  9. 2019Optimization for deep learning: theory and algorithmscited by 70
  10. 2018ProxQuant: Quantized Neural Networks via Proximal Operatorscited by 56
  11. 2017Deep Hyperspherical Learningcited by 55
  12. 2020On the distance between two neural networks and the stability of learningcited by 46
  13. 2018Decoupled Networkscited by 45
  14. 2020Orthogonal Over-Parameterized Trainingcited by 44
  15. 2016Non-Convex Projected Gradient Descent for Generalized Low-Rank Tensor Regressioncited by 23
  16. 2020Learning compositional functions via multiplicative weight updatescited by 20
  17. 2018Minnorm training: an algorithm for training over-parameterized deep neural networkscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 6; 2023: 6; 2024: 3; 2025: 0; 2026: 5.

23 papers.

Sort:

By area:

  1. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 396
  2. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  3. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 134
  4. 2023A Spectral Condition for Feature Learningcited by 101
  5. 2024nGPT: Normalized Transformer with Representation Learning on the Hyperspherecited by 68
  6. 2023Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networkscited by 50
  7. 2022LyaNet: A Lyapunov Framework for Training Neural ODEscited by 43
  8. 2024Deconstructing What Makes a Good Optimizer for Language Modelscited by 40
  9. 2021Why flatness does and does not correlate with generalization for deep neural networkscited by 13
  10. 2022Soft Augmentation for Image Classificationcited by 11
  11. 2022Investigating Generalization by Controlling Normalized Margincited by 5
  12. 2021Computing the Information Content of Trained Neural Networkscited by 3
11 more
  1. 2026Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectorscited by 3
  2. 2022Optimisation & Generalisation in Networks of Neuronscited by 2
  3. 2021Kernel Interpolation as a Bayes Point Machinecited by 1
  4. 2023Multiplicative update rules for accelerating deep learning training and increasing robustnesscited by 1
  5. 2023Design Of Rubble Analyzer Probe Using ML For Earthquakecited by 1
  6. 2023Automatic Optimisation of Normalised Neural Networkscited by 1
  7. 2026Demystifying Manifold Constraints in LLM Pre-trainingcited by 1
  8. 2022Towards understanding deep learning with the natural clustering priorcited by 0
  9. 2026Excitation: Momentum For Expertscited by 0
  10. 2026Training nGPTcited by 0
  11. 2026RODE: A Radial-Orthogonal Decoupled Engine for Optimizationcited by 0