# Placing Degree Scales After LayerNorm

arXiv: https://arxiv.org/abs/2606.14022 · HTML: https://mlsanity.com/paper/2606.14022/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 39 of its 45 references that are in the map.
- First posted on arXiv 12 Jun 2026.

## Builds on (most cited first, 30 of 39)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [How Attentive are Graph Attention Networks?](https://mlsanity.com/paper/2105.14491/) (2021)
- [TUDataset: A collection of benchmark datasets for learning with graphs](https://mlsanity.com/paper/2007.08663/) (2020)
- [Recipe for a General, Powerful, Scalable Graph Transformer](https://mlsanity.com/paper/2205.12454/) (2022)
- [Attention, Learn to Solve Routing Problems!](https://mlsanity.com/paper/1803.08475/) (2018)
- [Learning Combinatorial Optimization Algorithms over Graphs](https://mlsanity.com/paper/1704.01665/) (2017)
- [Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks](https://mlsanity.com/paper/1602.07868/) (2016)
- [Principal Neighbourhood Aggregation for Graph Nets](https://mlsanity.com/paper/2004.05718/) (2020)
- [Improving Graph Neural Network Expressivity via Subgraph Isomorphism Counting](https://mlsanity.com/paper/2006.09252/) (2020)
- [PairNorm: Tackling Oversmoothing in GNNs](https://mlsanity.com/paper/1909.12223/) (2019)
- [Using the Output Embedding to Improve Language Models](https://mlsanity.com/paper/1608.05859/) (2016)
- [Combinatorial optimization and reasoning with graph neural networks](https://mlsanity.com/paper/2102.09544/) (2021)
- [Random Features Strengthen Graph Neural Networks](https://mlsanity.com/paper/2002.03155/) (2020)
- [L2 Regularization versus Batch and Weight Normalization](https://mlsanity.com/paper/1706.05350/) (2017)
- [GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training](https://mlsanity.com/paper/2009.03294/) (2020)
- [Erdos Goes Neural: an Unsupervised Learning Framework for Combinatorial Optimization on Graphs](https://mlsanity.com/paper/2006.10643/) (2020)
- [Towards Deeper Graph Neural Networks with Differentiable Group Normalization](https://mlsanity.com/paper/2006.06972/) (2020)
- [Combinatorial Optimization with Physics-Inspired Graph Neural Networks](https://mlsanity.com/paper/2107.01188/) (2021)
- [Learning the Travelling Salesperson Problem Requires Rethinking Generalization](https://mlsanity.com/paper/2006.07054/) (2020)
- [Theoretical Analysis of Auto Rate-Tuning by Batch Normalization](https://mlsanity.com/paper/1812.03981/) (2018)
- [DEMO-Net: Degree-specific Graph Neural Networks for Node and Graph Classification](https://mlsanity.com/paper/1906.02319/) (2019)
- [Normalization and effective learning rates in reinforcement learning](https://mlsanity.com/paper/2407.01800/) (2024)
- [Understanding and Resolving Performance Degradation in Graph Convolutional Networks](https://mlsanity.com/paper/2006.07107/) (2020)

## Abstract (arXiv)

Graph neural networks (GNNs) are widely used to learn node-selection policies on graphs, and most stack graph attention (GAT) blocks with LayerNorm. On degree-sensitive tasks, LayerNorm tends to remove the degree signal these models need to rank nodes. Much recent work addresses this by redesigning normalizers or aggregators, which changes what these components compute but does not ask where, relative to LayerNorm, a degree scale should be applied. In this paper, we show that the answer follows from a single algebraic fact about LayerNorm. When a positive per-node scale is applied before LayerNorm, LayerNorm divides it out, and it never reaches the model's output. Applied after LayerNorm, the same scale comes through and reaches the score head as magnitude. From this placement rule we derive PostDeg, a parameter-free inverse-degree scale that we add as the single change to a fixed GAT backbone. PostDeg multiplies each node's normalized representation by an inverse function of its degree, and we compare it against controls in the same position. At every evaluation size, PostDeg improves over the LayerNorm backbone on influence maximization, network dismantling, and maximum independent set, and these controls show where the improvement comes from. The same scale before LayerNorm stays at the backbone, as the absorption identity predicts, and a constant scale after LayerNorm stays there too on all but the most heavy-tailed graphs, so the effect needs both the position after LayerNorm and a degree-dependent scale. The exact form of the scale matters much less, so we recommend PostDeg, which needs no tuning.