mlsanityData to 7 Oct 2026

Riemannian approach to batch normalization

arXiv 1709.09603 ↗First posted 27 Sep 2017cs.LGcited by 35

Not on the map. Why?

Abstract (arXiv)

Batch Normalization (BN) has proven to be an effective algorithm for deep neural network training by normalizing the input to each neuron and reducing the internal covariate shift. The space of weight vectors in the BN layer can be naturally interpreted as a Riemannian manifold, which is invariant to linear scaling of weights. Following the intrinsic geometry of this manifold provides a new learning rule that is more efficient and easier to analyze. We also propose intuitive and effective gradient clipping and regularization methods for the proposed algorithm by utilizing the geometry of the manifold. The resulting algorithm consistently outperforms the original BN on various types of network architectures and datasets.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Sep 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 4; 2023: 7; 2024: 6; 2025: 3; 2026: 4.

35 papers.

Sort:

By area:

  1. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 396
  2. 2021Large-Scale Differentially Private BERTcited by 117
  3. 2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 47
  4. 2022MAtt: A Manifold Attention Network for EEG Decodingcited by 43
  5. 2024Scale Equivariant Graph Metanetworkscited by 23
  6. 2021On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decaycited by 20
  7. 2022Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimescited by 19
  8. 2021Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependencecited by 12
  9. 2021Batch Normalization Preconditioning for Neural Network Trainingcited by 5
  10. 2022Riemannian Natural Gradient Methodscited by 5
  11. 2024Federated Learning on Riemannian Manifolds with Differential Privacycited by 5
  12. 2021Accelerating Training of Batch Normalization: A Manifold Perspectivecited by 4
23 more
  1. 2021Contextual Gradient Scaling for Few-Shot Learningcited by 4
  2. 2023Exploring the Complexity of Deep Neural Networks through Functional Equivalencecited by 4
  3. 2021BN-invariant sharpness regularizes the training model to better generalizationcited by 3
  4. 2021Tensor Normalization and Full Distribution Trainingcited by 3
  5. 2024Accelerating hyperbolic t-SNEcited by 3
  6. 2023A Block Coordinate Descent Method for Nonsmooth Composite Optimization under Orthogonality Constraintscited by 2
  7. 2025Directional Textual Inversion for Personalized Text-to-Image Generationcited by 2
  8. 2021Weight Rescaling: Effective and Robust Regularization for Deep Neural Networks with Batch Normalizationcited by 1
  9. 2021A Riemannian Mean Field Formulation for Two-layer Neural Networks with Batch Normalizationcited by 1
  10. 2023Decentralized Riemannian natural gradient methods with Kronecker-product approximationscited by 1
  11. 2023A survey of manifold learning and its applications for multimediacited by 1
  12. 2024Block-Diagonal Orthogonal Relation and Matrix Entity for Knowledge Graph Embeddingcited by 1
  13. 2024Learning-Rate-Free Stochastic Optimization over Riemannian Manifoldscited by 1
  14. 2022Auto-Encoding Goodness of Fitcited by 0
  15. 2023Faster Riemannian Newton-type Optimization by Subsampling and Cubic Regularizationcited by 0
  16. 2023Combining Primal and Dual Representations in Deep Restricted Kernel Machines Classifierscited by 0
  17. 2024Continuous-time Riemannian SGD and SVRG Flows on Wasserstein Probabilistic Spacecited by 0
  18. 2025GCSAM: Gradient Centralized Sharpness Aware Minimizationcited by 0
  19. 2025Can Stationary Distributions of Scale-Invariant Neural Networks Be Described by the Thermodynamics of an Ideal Gas?cited by 0
  20. 2026Retraction-Free Optimization over the Stiefel Manifold for the LoRA Fine-Tuningcited by 0
  21. 2026Training nGPTcited by 0
  22. 2026When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decaycited by 0
  23. 2026Continuous Optimization for p-adic Modelscited by 0