mlsanityData to 7 Oct 2026

Placing Degree Scales After LayerNorm

arXiv 2606.14022 ↗First posted 12 Jun 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Graph neural networks (GNNs) are widely used to learn node-selection policies on graphs, and most stack graph attention (GAT) blocks with LayerNorm. On degree-sensitive tasks, LayerNorm tends to remove the degree signal these models need to rank nodes. Much recent work addresses this by redesigning normalizers or aggregators, which changes what these components compute but does not ask where, relative to LayerNorm, a degree scale should be applied. In this paper, we show that the answer follows from a single algebraic fact about LayerNorm. When a positive per-node scale is applied before LayerNorm, LayerNorm divides it out, and it never reaches the model's output. Applied after LayerNorm, the same scale comes through and reaches the score head as magnitude. From this placement rule we derive PostDeg, a parameter-free inverse-degree scale that we add as the single change to a fixed GAT backbone. PostDeg multiplies each node's normalized representation by an inverse function of its degree, and we compare it against controls in the same position. At every evaluation size, PostDeg improves over the LayerNorm backbone on influence maximization, network dismantling, and maximum independent set, and these controls show where the improvement comes from. The same scale before LayerNorm stays at the backbone, as the absorption identity predicts, and a constant scale after LayerNorm stays there too on all but the most heavy-tailed graphs, so the effect needs both the position after LayerNorm and a degree-dependent scale. The exact form of the scale matters much less, so we recommend PostDeg, which needs no tuning.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 39 of its 45 references that are in the map. First posted on arXiv 12 Jun 2026.

Builds on

39 of 45 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  3. 2017Graph Attention Networkscited by 8,854
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2017Inductive Representation Learning on Large Graphscited by 6,284
  6. 2019Root Mean Square Layer Normalizationcited by 1,425
  7. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
  8. 2020On Layer Normalization in the Transformer Architecturecited by 906
  9. 2021How Attentive are Graph Attention Networks?cited by 818
  10. 2020TUDataset: A collection of benchmark datasets for learning with graphscited by 718
  11. 2022Recipe for a General, Powerful, Scalable Graph Transformercited by 702
  12. 2018Attention, Learn to Solve Routing Problems!cited by 646
27 more
  1. 2017Learning Combinatorial Optimization Algorithms over Graphscited by 557
  2. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  3. 2020Principal Neighbourhood Aggregation for Graph Netscited by 477
  4. 2020Improving Graph Neural Network Expressivity via Subgraph Isomorphism Countingcited by 312
  5. 2019PairNorm: Tackling Oversmoothing in GNNscited by 297
  6. 2021Combinatorial optimization and reasoning with graph neural networkscited by 267
  7. 2016Using the Output Embedding to Improve Language Modelscited by 266
  8. 2020Random Features Strengthen Graph Neural Networkscited by 184
  9. 2017L2 Regularization versus Batch and Weight Normalizationcited by 121
  10. 2020GraphNorm: A Principled Approach to Accelerating Graph Neural Network Trainingcited by 116
  11. 2020Erdos Goes Neural: an Unsupervised Learning Framework for Combinatorial Optimization on Graphscited by 115
  12. 2020Towards Deeper Graph Neural Networks with Differentiable Group Normalizationcited by 113
  13. 2021Combinatorial Optimization with Physics-Inspired Graph Neural Networkscited by 107
  14. 2020Learning the Travelling Salesperson Problem Requires Rethinking Generalizationcited by 96
  15. 2018Theoretical Analysis of Auto Rate-Tuning by Batch Normalizationcited by 65
  16. 2019DEMO-Net: Degree-specific Graph Neural Networks for Node and Graph Classificationcited by 59
  17. 2024Normalization and effective learning rates in reinforcement learningcited by 55
  18. 2020Understanding and Resolving Performance Degradation in Graph Convolutional Networkscited by 53
  19. 2019Understanding the Representation Power of Graph Neural Networks in Learning Graph Topologycited by 52
  20. 2022Modern graph neural networks do worse than classical greedy algorithms in solving combinatorial optimization problems like maximum independent setcited by 33
  21. 2023Deep Graph Representation Learning and Optimization for Influence Maximizationcited by 27
  22. 2020Learning Graph Normalization for Graph Neural Networkscited by 16
  23. 2022ResNorm: Tackling Long-tailed Degree Distribution Issue in Graph Neural Networks via Normalizationcited by 15
  24. 2016Network dismantlingcited by 12
  25. 2022ToupleGDD: A Fine-Designed Solution of Influence Maximization by Deep Reinforcement Learningcited by 11
  26. 2024GRANOLA: Adaptive Normalization for Graph Neural Networkscited by 7
  27. 2026Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Modelscited by 3

Built on by

Not cited yet.