mlsanityData to 14 Oct 2026

A Riemannian Mean Field Formulation for Two-layer Neural Networks with Batch Normalization

arXiv 2110.08725 ↗First posted 17 Oct 2021cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

The training dynamics of two-layer neural networks with batch normalization (BN) is studied. It is written as the training dynamics of a neural network without BN on a Riemannian manifold. Therefore, we identify BN's effect of changing the metric in the parameter space. Later, the infinite-width limit of the two-layer neural networks with BN is considered, and a mean-field formulation is derived for the training dynamics. The training dynamics of the mean-field formulation is shown to be the Wasserstein gradient flow on the manifold. Theoretical analysis are provided on the well-posedness and convergence of the Wasserstein gradient flow.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 16 of its 20 references that are in the map. First posted on arXiv 17 Oct 2021.

Builds on

16 of 20 references.

References · 16

  1. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  2. 2016Layer Normalizationcited by 5,669
  3. 2018Group Normalizationcited by 1,524
  4. 2017Self-Normalizing Neural Networkscited by 598
  5. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 543
  6. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  7. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  8. 2018How Does Batch Normalization Help Optimization?cited by 357
  9. 2018Understanding Batch Normalizationcited by 159
  10. 2018Theoretical Analysis of Auto Rate-Tuning by Batch Normalizationcited by 65
  11. 2018Towards Understanding Regularization in Batch Normalizationcited by 56
  12. 2019Machine Learning from a Continuous Viewpointcited by 50
4 more
  1. 2020Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networkscited by 44
  2. 2017Riemannian approach to batch normalizationcited by 35
  3. 2019Mean Field Limit of the Learning Dynamics of Multilayer Neural Networkscited by 25
  4. 2019Mean-field Analysis of Batch Normalizationcited by 3

Built on by

Cited by year

Citing papers per year: 2022: 1.

1 paper.

  1. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86