# The Normalization Method for Alleviating Pathological Sharpness in Wide Neural Networks

arXiv: https://arxiv.org/abs/1906.02926 · HTML: https://mlsanity.com/paper/1906.02926/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jun 2019.

## Built on by (most cited first, 20 of 20)

- [ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks](https://mlsanity.com/paper/2102.11600/) (2021)
- [Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction](https://mlsanity.com/paper/2206.07085/) (2022)
- [Transformers without Normalization](https://mlsanity.com/paper/2503.10622/) (2025)
- [Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stability](https://mlsanity.com/paper/2207.12678/) (2022)
- [Batch Normalization Orthogonalizes Representations in Deep Random Networks](https://mlsanity.com/paper/2106.03970/) (2021)
- [Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learning](https://mlsanity.com/paper/2106.05956/) (2021)
- [Stronger Normalization-Free Transformers](https://mlsanity.com/paper/2512.10938/) (2025)
- [On the Nonlinearity of Layer Normalization](https://mlsanity.com/paper/2406.01255/) (2024)
- [Asymptotic Freeness of Layerwise Jacobians Caused by Invariance of Multilayer Perceptron: The Haar Orthogonal Case](https://mlsanity.com/paper/2103.13466/) (2021)
- [Continuous-Time Analysis of Adaptive Optimization and Normalization](https://mlsanity.com/paper/2411.05746/) (2024)
- [TorchNTK: A Library for Calculation of Neural Tangent Kernels of PyTorch Models](https://mlsanity.com/paper/2205.12372/) (2022)
- [Beyond accuracy: generalization properties of bio-plausible temporal credit assignment rules](https://mlsanity.com/paper/2206.00823/) (2022)
- [CHAIN: Enhancing Generalization in Data-Efficient GANs via lipsCHitz continuity constrAIned Normalization](https://mlsanity.com/paper/2404.00521/) (2024)
- [Towards the Spectral bias Alleviation by Normalizations in Coordinate Networks](https://mlsanity.com/paper/2407.17834/) (2024)
- [Component-Wise Natural Gradient Descent -- An Efficient Neural Network Optimization](https://mlsanity.com/paper/2210.05268/) (2022)
- [Neuro-Visualizer: An Auto-encoder-based Loss Landscape Visualization Method](https://mlsanity.com/paper/2309.14601/) (2023)
- [Non-identifiability distinguishes Neural Networks among Parametric Models](https://mlsanity.com/paper/2504.18017/) (2025)
- [Parallel Layer Normalization for Universal Approximation](https://mlsanity.com/paper/2505.13142/) (2025)
- [Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learning](https://mlsanity.com/paper/2505.21877/) (2025)
- [The Hidden Power of Normalization Layers in Neural Networks: Exponential Capacity Control](https://mlsanity.com/paper/2511.00958/) (2025)

## Abstract (arXiv)

Normalization methods play an important role in enhancing the performance of deep learning while their theoretical understandings have been limited. To theoretically elucidate the effectiveness of normalization, we quantify the geometry of the parameter space determined by the Fisher information matrix (FIM), which also corresponds to the local shape of the loss landscape under certain conditions. We analyze deep neural networks with random initialization, which is known to suffer from a pathologically sharp shape of the landscape when the network becomes sufficiently wide. We reveal that batch normalization in the last layer contributes to drastically decreasing such pathological sharpness if the width and sample number satisfy a specific condition. In contrast, it is hard for batch normalization in the middle hidden layers to alleviate pathological sharpness in many settings. We also found that layer normalization cannot alleviate pathological sharpness either. Thus, we can conclude that batch normalization in the last layer significantly contributes to decreasing the sharpness induced by the FIM.