# Natural Neural Networks

arXiv: https://arxiv.org/abs/1507.00210 · HTML: https://mlsanity.com/paper/1507.00210/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jul 2015.

## Built on by (most cited first, 30 of 44)

- [Powerpropagation: A sparsity inducing weight reparameterisation](https://mlsanity.com/paper/2110.00296/) (2021)
- [Orthogonal Projection Loss](https://mlsanity.com/paper/2103.14021/) (2021)
- [Shapley Explanation Networks](https://mlsanity.com/paper/2104.02297/) (2021)
- [Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscape](https://mlsanity.com/paper/2201.08025/) (2022)
- [Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities](https://mlsanity.com/paper/2210.06640/) (2022)
- [Gradient Descent on Neurons and its Link to Approximate Second-Order Optimization](https://mlsanity.com/paper/2201.12250/) (2022)
- [Random initialisations performing above chance and how to find them](https://mlsanity.com/paper/2209.07509/) (2022)
- [VNE: An Effective Method for Improving Deep Representation by Manipulating Eigenvalue Distribution](https://mlsanity.com/paper/2304.01434/) (2023)
- [The Geometry of Neural Nets' Parameter Spaces Under Reparametrization](https://mlsanity.com/paper/2302.07384/) (2023)
- [Approximate Nearest Neighbor Search under Neural Similarity Metric for Large-Scale Recommendation](https://mlsanity.com/paper/2202.10226/) (2022)
- [Towards Quantifying the Hessian Structure of Neural Networks](https://mlsanity.com/paper/2505.02809/) (2025)
- [Geometry and convergence of natural policy gradient methods](https://mlsanity.com/paper/2211.02105/) (2022)
- [Second-Order Neural ODE Optimizer](https://mlsanity.com/paper/2109.14158/) (2021)
- [Stochastic Whitening Batch Normalization](https://mlsanity.com/paper/2106.04413/) (2021)
- [TENGraD: Time-Efficient Natural Gradient Descent with Exact Fisher-Block Inversion](https://mlsanity.com/paper/2106.03947/) (2021)
- [Delving into the Estimation Shift of Batch Normalization in a Network](https://mlsanity.com/paper/2203.10778/) (2022)
- [Efficient Deep Learning with Decorrelated Backpropagation](https://mlsanity.com/paper/2405.02385/) (2024)
- [Correlations Are Ruining Your Gradient Descent](https://mlsanity.com/paper/2407.10780/) (2024)
- [Budgeted Online Continual Learning by Adaptive Layer Freezing and Frequency-based Sampling](https://mlsanity.com/paper/2410.15143/) (2024)
- [Step-size Optimization for Continual Learning](https://mlsanity.com/paper/2401.17401/) (2024)
- [Gradient-Free Training of Recurrent Neural Networks using Random Perturbations](https://mlsanity.com/paper/2405.08967/) (2024)
- [Fast Deep Hedging with Second-Order Optimization](https://mlsanity.com/paper/2410.22568/) (2024)
- [Path-conditioned training: a principled way to rescale ReLU neural networks](https://mlsanity.com/paper/2602.19799/) (2026)
- [Batch Normalization Preconditioning for Neural Network Training](https://mlsanity.com/paper/2108.01110/) (2021)
- [Feedback Gradient Descent: Efficient and Stable Optimization with Orthogonality for DNNs](https://mlsanity.com/paper/2205.08385/) (2022)
- [Hierarchical Semantic Tree Concept Whitening for Interpretable Image Classification](https://mlsanity.com/paper/2307.04343/) (2023)
- [Improving Generalization of Batch Whitening by Convolutional Unit Optimization](https://mlsanity.com/paper/2108.10629/) (2021)
- [Exploiting Invariance in Training Deep Neural Networks](https://mlsanity.com/paper/2103.16634/) (2021)
- [Learnable Adaptive Cosine Estimator (LACE) for Image Classification](https://mlsanity.com/paper/2110.05324/) (2021)
- [Node Perturbation Can Effectively Train Multi-Layer Neural Networks](https://mlsanity.com/paper/2310.00965/) (2023)

## Abstract (arXiv)

We introduce Natural Neural Networks, a novel family of algorithms that speed up convergence by adapting their internal representation during training to improve conditioning of the Fisher matrix. In particular, we show a specific example that employs a simple and efficient reparametrization of the neural network weights by implicitly whitening the representation obtained at each layer, while preserving the feed-forward computation of the network. Such networks can be trained efficiently via the proposed Projected Natural Gradient Descent algorithm (PRONG), which amortizes the cost of these reparametrizations over many parameter updates and is closely related to the Mirror Descent online learning algorithm. We highlight the benefits of our method on both unsupervised and supervised learning tasks, and showcase its scalability by training on the large-scale ImageNet Challenge dataset.