# Automatic Optimisation of Normalised Neural Networks

arXiv: https://arxiv.org/abs/2312.10672 · HTML: https://mlsanity.com/paper/2312.10672/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 23 references that are in the map.
- First posted on arXiv 17 Dec 2023.

## Builds on (most cited first, 14 of 14)

- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Spectrally-normalized margin bounds for neural networks](https://mlsanity.com/paper/1706.08498/) (2017)
- [Sorting out Lipschitz function approximation](https://mlsanity.com/paper/1811.05381/) (2018)
- [Neural Lander: Stable Drone Landing Control using Learned Dynamics](https://mlsanity.com/paper/1811.08027/) (2018)
- [Neural-Fly Enables Rapid Learning for Agile Flight in Strong Winds](https://mlsanity.com/paper/2205.06908/) (2022)
- [Direct Parameterization of Lipschitz-Bounded Deep Networks](https://mlsanity.com/paper/2301.11526/) (2023)
- [On the distance between two neural networks and the stability of learning](https://mlsanity.com/paper/2002.03432/) (2020)
- [Robust Regression for Safe Exploration in Control](https://mlsanity.com/paper/1906.05819/) (2019)
- [Learning by Turning: Neural Architecture Aware Optimisation](https://mlsanity.com/paper/2102.07227/) (2021)
- [Why Spectral Normalization Stabilizes GANs: Analysis and Improvements](https://mlsanity.com/paper/2009.02773/) (2020)
- [Neural-Swarm2: Planning and Control of Heterogeneous Multirotor Swarms using Learned Interactions](https://mlsanity.com/paper/2012.05457/) (2020)
- [Spherical Perspective on Learning with Normalization Layers](https://mlsanity.com/paper/2006.13382/) (2020)
- [Bridging the Model-Reality Gap with Lipschitz Network Adaptation](https://mlsanity.com/paper/2112.03756/) (2021)
- [Optimisation & Generalisation in Networks of Neurons](https://mlsanity.com/paper/2210.10101/) (2022)

## Built on by (most cited first, 1 of 1)

- [Scalable Optimization in the Modular Norm](https://mlsanity.com/paper/2405.14813/) (2024)

## Abstract (arXiv)

We propose automatic optimisation methods considering the geometry of matrix manifold for the normalised parameters of neural networks. Layerwise weight normalisation with respect to Frobenius norm is utilised to bound the Lipschitz constant and to enhance gradient reliability so that the trained networks are suitable for control applications. Our approach first initialises the network and normalises the data with respect to the ℓ2-ℓ2 gain of the initialised network. Then, the proposed algorithms take the update structure based on the exponential map on high-dimensional spheres. Given an update direction such as that of the negative Riemannian gradient, we propose two different ways to determine the stepsize for descent. The first algorithm utilises automatic differentiation of the objective function along the update curve defined on the combined manifold of spheres. The directional second-order derivative information can be utilised without requiring explicit construction of the Hessian. The second algorithm utilises the majorisation-minimisation framework via architecture-aware majorisation for neural networks. With these new developments, the proposed methods avoid manual tuning and scheduling of the learning rate, thus providing an automated pipeline for optimizing normalised neural networks.