# Riemannian approach to batch normalization

arXiv: https://arxiv.org/abs/1709.09603 · HTML: https://mlsanity.com/paper/1709.09603/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Sep 2017.

## Built on by (most cited first, 30 of 35)

- [Analyzing and Improving the Training Dynamics of Diffusion Models](https://mlsanity.com/paper/2312.02696/) (2023)
- [Large-Scale Differentially Private BERT](https://mlsanity.com/paper/2108.01624/) (2021)
- [Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks](https://mlsanity.com/paper/2105.02716/) (2021)
- [MAtt: A Manifold Attention Network for EEG Decoding](https://mlsanity.com/paper/2210.01986/) (2022)
- [Scale Equivariant Graph Metanetworks](https://mlsanity.com/paper/2406.10685/) (2024)
- [On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decay](https://mlsanity.com/paper/2106.15739/) (2021)
- [Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimes](https://mlsanity.com/paper/2209.03695/) (2022)
- [Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence](https://mlsanity.com/paper/2106.03743/) (2021)
- [Batch Normalization Preconditioning for Neural Network Training](https://mlsanity.com/paper/2108.01110/) (2021)
- [Riemannian Natural Gradient Methods](https://mlsanity.com/paper/2207.07287/) (2022)
- [Federated Learning on Riemannian Manifolds with Differential Privacy](https://mlsanity.com/paper/2404.10029/) (2024)
- [Accelerating Training of Batch Normalization: A Manifold Perspective](https://mlsanity.com/paper/2101.02916/) (2021)
- [Contextual Gradient Scaling for Few-Shot Learning](https://mlsanity.com/paper/2110.10353/) (2021)
- [Exploring the Complexity of Deep Neural Networks through Functional Equivalence](https://mlsanity.com/paper/2305.11417/) (2023)
- [BN-invariant sharpness regularizes the training model to better generalization](https://mlsanity.com/paper/2101.02944/) (2021)
- [Tensor Normalization and Full Distribution Training](https://mlsanity.com/paper/2109.02345/) (2021)
- [Accelerating hyperbolic t-SNE](https://mlsanity.com/paper/2401.13708/) (2024)
- [A Block Coordinate Descent Method for Nonsmooth Composite Optimization under Orthogonality Constraints](https://mlsanity.com/paper/2304.03641/) (2023)
- [Directional Textual Inversion for Personalized Text-to-Image Generation](https://mlsanity.com/paper/2512.13672/) (2025)
- [Weight Rescaling: Effective and Robust Regularization for Deep Neural Networks with Batch Normalization](https://mlsanity.com/paper/2102.03497/) (2021)
- [A Riemannian Mean Field Formulation for Two-layer Neural Networks with Batch Normalization](https://mlsanity.com/paper/2110.08725/) (2021)
- [Decentralized Riemannian natural gradient methods with Kronecker-product approximations](https://mlsanity.com/paper/2303.09611/) (2023)
- [A survey of manifold learning and its applications for multimedia](https://mlsanity.com/paper/2310.12986/) (2023)
- [Block-Diagonal Orthogonal Relation and Matrix Entity for Knowledge Graph Embedding](https://mlsanity.com/paper/2401.05967/) (2024)
- [Learning-Rate-Free Stochastic Optimization over Riemannian Manifolds](https://mlsanity.com/paper/2406.02296/) (2024)
- [GCSAM: Gradient Centralized Sharpness Aware Minimization](https://mlsanity.com/paper/2501.11584/) (2025)
- [Auto-Encoding Goodness of Fit](https://mlsanity.com/paper/2210.06546/) (2022)
- [Faster Riemannian Newton-type Optimization by Subsampling and Cubic Regularization](https://mlsanity.com/paper/2302.11076/) (2023)
- [Combining Primal and Dual Representations in Deep Restricted Kernel Machines Classifiers](https://mlsanity.com/paper/2306.07015/) (2023)
- [Continuous-time Riemannian SGD and SVRG Flows on Wasserstein Probabilistic Space](https://mlsanity.com/paper/2401.13530/) (2024)

## Abstract (arXiv)

Batch Normalization (BN) has proven to be an effective algorithm for deep neural network training by normalizing the input to each neuron and reducing the internal covariate shift. The space of weight vectors in the BN layer can be naturally interpreted as a Riemannian manifold, which is invariant to linear scaling of weights. Following the intrinsic geometry of this manifold provides a new learning rule that is more efficient and easier to analyze. We also propose intuitive and effective gradient clipping and regularization methods for the proposed algorithm by utilizing the geometry of the manifold. The resulting algorithm consistently outperforms the original BN on various types of network architectures and datasets.