# GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training

arXiv: https://arxiv.org/abs/2009.03294 · HTML: https://mlsanity.com/paper/2009.03294/

## Key facts

- Cited by 117 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Sep 2020.

## Built on by (most cited first, 30 of 117)

- [Do Transformers Really Perform Bad for Graph Representation?](https://mlsanity.com/paper/2106.05234/) (2021)
- [Training Graph Neural Networks with 1000 Layers](https://mlsanity.com/paper/2106.07476/) (2021)
- [Classic GNNs are Strong Baselines: Reassessing GNNs for Node Classification](https://mlsanity.com/paper/2406.08993/) (2024)
- [FAENet: Frame Averaging Equivariant GNN for Materials Modeling](https://mlsanity.com/paper/2305.05577/) (2023)
- [GRPE: Relative Positional Encoding for Graph Transformer](https://mlsanity.com/paper/2201.12787/) (2022)
- [Efficient 3D Semantic Segmentation with Superpoint Transformer](https://mlsanity.com/paper/2306.08045/) (2023)
- [Optimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depth](https://mlsanity.com/paper/2105.04550/) (2021)
- [Raising the Bar in Graph-level Anomaly Detection](https://mlsanity.com/paper/2205.13845/) (2022)
- [Accelerating Large Scale Real-Time GNN Inference using Channel Pruning](https://mlsanity.com/paper/2105.04528/) (2021)
- [Latent Graph Representations for Critical View of Safety Assessment](https://mlsanity.com/paper/2212.04155/) (2022)
- [Residual Connections and Normalization Can Provably Prevent Oversmoothing in GNNs](https://mlsanity.com/paper/2406.02997/) (2024)
- [Scalable Discrete Diffusion Samplers: Combinatorial Optimization and Statistical Physics](https://mlsanity.com/paper/2502.08696/) (2025)
- [Can Classic GNNs Be Strong Baselines for Graph-level Tasks? Simple Architectures Meet Excellence](https://mlsanity.com/paper/2502.09263/) (2025)
- [A New Perspective on the Effects of Spectrum in Graph Neural Networks](https://mlsanity.com/paper/2112.07160/) (2021)
- [Molecular Representation Learning via Heterogeneous Motif Graph Neural Networks](https://mlsanity.com/paper/2202.00529/) (2022)
- [Neural incomplete factorization: learning preconditioners for the conjugate gradient method](https://mlsanity.com/paper/2305.16368/) (2023)
- [Set Norm and Equivariant Skip Connections: Putting the Deep in Deep Sets](https://mlsanity.com/paper/2206.11925/) (2022)
- [Weisfeiler and Leman Go Infinite: Spectral and Combinatorial Pre-Colorings](https://mlsanity.com/paper/2201.13410/) (2022)
- [ResNorm: Tackling Long-tailed Degree Distribution Issue in Graph Neural Networks via Normalization](https://mlsanity.com/paper/2206.08181/) (2022)
- [ZigZag: Universal Sampling-free Uncertainty Estimation Through Two-Step Inference](https://mlsanity.com/paper/2211.11435/) (2022)
- [MuSe-GNN: Learning Unified Gene Representation From Multimodal Biological Graph Data](https://mlsanity.com/paper/2310.02275/) (2023)
- [Efficiently predicting high resolution mass spectra with graph neural networks](https://mlsanity.com/paper/2301.11419/) (2023)
- [Convergent Graph Solvers](https://mlsanity.com/paper/2106.01680/) (2021)
- [RSC: Accelerating Graph Neural Networks Training via Randomized Sparse Computations](https://mlsanity.com/paper/2210.10737/) (2022)
- [Gibbs-Helmholtz Graph Neural Network: capturing the temperature dependency of activity coefficients at infinite dilution](https://mlsanity.com/paper/2212.01199/) (2022)
- [Improving Expressivity of GNNs with Subgraph-specific Factor Embedded Normalization](https://mlsanity.com/paper/2305.19903/) (2023)
- [From Local to Global: Spectral-Inspired Graph Neural Networks](https://mlsanity.com/paper/2209.12054/) (2022)
- [Connectivity Optimized Nested Graph Networks for Crystal Structures](https://mlsanity.com/paper/2302.14102/) (2023)
- [An end-to-end attention-based approach for learning on graphs](https://mlsanity.com/paper/2402.10793/) (2024)
- [Are GATs Out of Balance?](https://mlsanity.com/paper/2310.07235/) (2023)

## Abstract (arXiv)

Normalization is known to help the optimization of deep neural networks. Curiously, different architectures require specialized normalization methods. In this paper, we study what normalization is effective for Graph Neural Networks (GNNs). First, we adapt and evaluate the existing methods from other domains to GNNs. Faster convergence is achieved with InstanceNorm compared to BatchNorm and LayerNorm. We provide an explanation by showing that InstanceNorm serves as a preconditioner for GNNs, but such preconditioning effect is weaker with BatchNorm due to the heavy batch noise in graph datasets. Second, we show that the shift operation in InstanceNorm results in an expressiveness degradation of GNNs for highly regular graphs. We address this issue by proposing GraphNorm with a learnable shift. Empirically, GNNs with GraphNorm converge faster compared to GNNs using other normalization. GraphNorm also improves the generalization of GNNs, achieving better performance on graph classification benchmarks.