# On Symmetry and Initialization for Neural Networks

arXiv: https://arxiv.org/abs/1907.00560 · HTML: https://mlsanity.com/paper/1907.00560/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jul 2019.

## Built on by (most cited first, 5 of 5)

- [Improving Convergence and Generalization Using Parameter Symmetries](https://mlsanity.com/paper/2305.13404/) (2023)
- [Fantastic Generalization Measures are Nowhere to be Found](https://mlsanity.com/paper/2309.13658/) (2023)
- [On the non-universality of deep learning: quantifying the cost of symmetry](https://mlsanity.com/paper/2208.03113/) (2022)
- [Hardness of Learning Fixed Parities with Neural Networks](https://mlsanity.com/paper/2501.00817/) (2025)
- [Symmetry and Generalisation in Neural Approximations of Renormalisation Transformations](https://mlsanity.com/paper/2510.16591/) (2025)

## Abstract (arXiv)

This work provides an additional step in the theoretical understanding of neural networks. We consider neural networks with one hidden layer and show that when learning symmetric functions, one can choose initial conditions so that standard SGD training efficiently produces generalization guarantees. We empirically verify this and show that this does not hold when the initial conditions are chosen at random. The proof of convergence investigates the interaction between the two layers of the network. Our results highlight the importance of using symmetry in the design of neural networks.