mlsanityData to 7 Oct 2026

Natural Neural Networks

arXiv 1507.00210 ↗First posted 1 Jul 2015stat.ML cs.LG cs.NEcited by 44

Not on the map. Why?

Abstract (arXiv)

We introduce Natural Neural Networks, a novel family of algorithms that speed up convergence by adapting their internal representation during training to improve conditioning of the Fisher matrix. In particular, we show a specific example that employs a simple and efficient reparametrization of the neural network weights by implicitly whitening the representation obtained at each layer, while preserving the feed-forward computation of the network. Such networks can be trained efficiently via the proposed Projected Natural Gradient Descent algorithm (PRONG), which amortizes the cost of these reparametrizations over many parameter updates and is closely related to the Mirror Descent online learning algorithm. We highlight the benefits of our method on both unsupervised and supervised learning tasks, and showcase its scalability by training on the large-scale ImageNet Challenge dataset.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Jul 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 10; 2023: 5; 2024: 6; 2025: 5; 2026: 6.

44 papers.

Sort:

  1. 2021Powerpropagation: A sparsity inducing weight reparameterisationcited by 49
  2. 2021Orthogonal Projection Losscited by 44
  3. 2021Shapley Explanation Networkscited by 41
  4. 2022Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscapecited by 39
  5. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  6. 2022Gradient Descent on Neurons and its Link to Approximate Second-Order Optimizationcited by 29
  7. 2022Random initialisations performing above chance and how to find themcited by 23
  8. 2023VNE: An Effective Method for Improving Deep Representation by Manipulating Eigenvalue Distributioncited by 17
  9. 2023The Geometry of Neural Nets' Parameter Spaces Under Reparametrizationcited by 16
  10. 2022Approximate Nearest Neighbor Search under Neural Similarity Metric for Large-Scale Recommendationcited by 14
  11. 2025Towards Quantifying the Hessian Structure of Neural Networkscited by 13
  12. 2022Geometry and convergence of natural policy gradient methodscited by 12
32 more
  1. 2021Second-Order Neural ODE Optimizercited by 11
  2. 2021Stochastic Whitening Batch Normalizationcited by 9
  3. 2021TENGraD: Time-Efficient Natural Gradient Descent with Exact Fisher-Block Inversioncited by 7
  4. 2022Delving into the Estimation Shift of Batch Normalization in a Networkcited by 7
  5. 2024Efficient Deep Learning with Decorrelated Backpropagationcited by 7
  6. 2024Correlations Are Ruining Your Gradient Descentcited by 7
  7. 2024Budgeted Online Continual Learning by Adaptive Layer Freezing and Frequency-based Samplingcited by 7
  8. 2024Step-size Optimization for Continual Learningcited by 6
  9. 2024Gradient-Free Training of Recurrent Neural Networks using Random Perturbationscited by 6
  10. 2024Fast Deep Hedging with Second-Order Optimizationcited by 6
  11. 2026Path-conditioned training: a principled way to rescale ReLU neural networkscited by 6
  12. 2021Batch Normalization Preconditioning for Neural Network Trainingcited by 5
  13. 2022Feedback Gradient Descent: Efficient and Stable Optimization with Orthogonality for DNNscited by 5
  14. 2023Hierarchical Semantic Tree Concept Whitening for Interpretable Image Classificationcited by 5
  15. 2021Improving Generalization of Batch Whitening by Convolutional Unit Optimizationcited by 3
  16. 2021Exploiting Invariance in Training Deep Neural Networkscited by 2
  17. 2021Learnable Adaptive Cosine Estimator (LACE) for Image Classificationcited by 2
  18. 2023Node Perturbation Can Effectively Train Multi-Layer Neural Networkscited by 2
  19. 2021A Distributed Optimisation Framework Combining Natural Gradient with Hessian-Free for Discriminative Sequence Trainingcited by 1
  20. 2021Information Geometry and Classical Cram'er-Rao Type Inequalitiescited by 1
  21. 2022What do CNNs Learn in the First Layer and Why? A Linear Systems Perspectivecited by 1
  22. 2025Hindsight-Guided Momentum (HGM) Optimizer: An Approach to Adaptive Learning Ratecited by 1
  23. 2025IBNorm: Information-Bottleneck Inspired Normalization for Representation Learningcited by 1
  24. 2026On the Convergence Behavior of Preconditioned Gradient Descent Toward the Rich Learning Regimecited by 1
  25. 2026Conditioned Direct Feedback Alignment via Activity and Error Geometrycited by 1
  26. 2022GRM: Gradient Rectification Module for Visual Place Retrievalcited by 0
  27. 2023Jorge: Approximate Preconditioning for GPU-efficient Second-order Optimizationcited by 0
  28. 2025Decorrelation Speeds Up Vision Transformerscited by 0
  29. 2025A Unified Perspective on Optimization in Machine Learning and Neuroscience: From Gradient Descent to Neural Adaptationcited by 0
  30. 2026Riemannian Optimization in Modular Systemscited by 0
  31. 2026Natural Gradient Descent for Online Continual Learningcited by 0
  32. 2026Token Geometrycited by 0