# Symmetries in Overparametrized Neural Networks: A Mean-Field View arXiv: https://arxiv.org/abs/2405.19995 · HTML: https://mlsanity.com/paper/2405.19995/ ## Key facts - Cited by 2 papers in the map (data through 14 Oct 2026). - Builds on 34 of its 60 references that are in the map. - First posted on arXiv 30 May 2024. ## Builds on (most cited first, 30 of 34) - [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017) - [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021) - [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016) - [A Mean Field View of the Landscape of Two-Layers Neural Networks](https://mlsanity.com/paper/1804.06561/) (2018) - [General E(2)-Equivariant Steerable CNNs](https://mlsanity.com/paper/1911.08251/) (2019) - [On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport](https://mlsanity.com/paper/1805.09545/) (2018) - [On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups](https://mlsanity.com/paper/1802.03690/) (2018) - [A General Theory of Equivariant CNNs on Homogeneous Spaces](https://mlsanity.com/paper/1811.02017/) (2018) - [Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit](https://mlsanity.com/paper/1902.06015/) (2019) - [A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Groups](https://mlsanity.com/paper/2104.09459/) (2021) - [On the Universality of Invariant Networks](https://mlsanity.com/paper/1901.09342/) (2019) - [Equivariance Through Parameter-Sharing](https://mlsanity.com/paper/1702.08389/) (2017) - [Universal approximations of invariant maps by neural networks](https://mlsanity.com/paper/1804.10306/) (2018) - [Mean Field Analysis of Neural Networks: A Central Limit Theorem](https://mlsanity.com/paper/1808.09372/) (2018) - [A Kernel Theory of Modern Data Augmentation](https://mlsanity.com/paper/1803.06084/) (2018) - [Trainability and Accuracy of Neural Networks: An Interacting Particle System Approach](https://mlsanity.com/paper/1805.00915/) (2018) - [Provably Strict Generalisation Benefit for Equivariant Models](https://mlsanity.com/paper/2102.10333/) (2021) - [Learning with invariances in random features and kernel models](https://mlsanity.com/paper/2102.13219/) (2021) - [On the Benefits of Invariance in Neural Networks](https://mlsanity.com/paper/2005.00178/) (2020) - [Enhanced Convolutional Neural Tangent Kernels](https://mlsanity.com/paper/1911.00809/) (2019) - [Mean-Field Langevin Dynamics: Exponential Convergence and Annealing](https://mlsanity.com/paper/2202.01009/) (2022) - [Convex Analysis of the Mean Field Langevin Dynamics](https://mlsanity.com/paper/2201.10469/) (2022) - [Mean-Field Langevin Dynamics and Energy Landscape of Neural Networks](https://mlsanity.com/paper/1905.07769/) (2019) - [Universal Equivariant Multilayer Perceptrons](https://mlsanity.com/paper/2002.02912/) (2020) - [A Wigner-Eckart Theorem for Group Equivariant Convolution Kernels](https://mlsanity.com/paper/2010.10952/) (2020) - [Approximation-Generalization Trade-offs under (Approximate) Group Equivariance](https://mlsanity.com/paper/2305.17592/) (2023) - [A Dynamical Central Limit Theorem for Shallow Neural Networks](https://mlsanity.com/paper/2008.09623/) (2020) - [Homogeneous vector bundles and G-equivariant convolutional neural networks](https://mlsanity.com/paper/2105.05400/) (2021) - [Quantitative Propagation of Chaos for SGD in Wide Neural Networks](https://mlsanity.com/paper/2007.06352/) (2020) - [Convergence of mean-field Langevin dynamics: Time and space discretization, stochastic gradient, and variance reduction](https://mlsanity.com/paper/2306.07221/) (2023) ## Built on by (most cited first, 2 of 2) - [Ensembles provably learn equivariance through data augmentation](https://mlsanity.com/paper/2410.01452/) (2024) - [Data Augmentation and Regularization for Learning Group Equivariance](https://mlsanity.com/paper/2502.06547/) (2025) ## Abstract (arXiv) We develop a Mean-Field (MF) view of the learning dynamics of overparametrized Artificial Neural Networks (NN) under data symmetric in law wrt the action of a general compact group G. We consider for this a class of generalized shallow NNs given by an ensemble of N multi-layer units, jointly trained using stochastic gradient descent (SGD) and possibly symmetry-leveraging (SL) techniques, such as Data Augmentation (DA), Feature Averaging (FA) or Equivariant Architectures (EA). We introduce the notions of weakly and strongly invariant laws (WI and SI) on the parameter space of each single unit, corresponding, respectively, to G-invariant distributions, and to distributions supported on parameters fixed by the group action (which encode EA). This allows us to define symmetric models compatible with taking N→∞ and give an interpretation of the asymptotic dynamics of DA, FA and EA in terms of Wasserstein Gradient Flows describing their MF limits. When activations respect the group action, we show that, for symmetric data, DA, FA and freely-trained models obey the exact same MF dynamic, which stays in the space of WI laws and minimizes therein the population risk. We also give a counterexample to the general attainability of an optimum over SI laws. Despite this, quite remarkably, we show that the set of SI laws is also preserved by the MF dynamics even when freely trained. This sharply contrasts the finite-N setting, in which EAs are generally not preserved by unconstrained SGD. We illustrate the validity of our findings as N gets larger in a teacher-student experimental setting, training a student NN to learn from a WI, SI or arbitrary teacher model through various SL schemes. We last deduce a data-driven heuristic to discover the largest subspace of parameters supporting SI distributions for a problem, that could be used for designing EA with minimal generalization error.