Graph Neural Networks Need Cluster-Normalize-Activate Modules
Not on the map. Why?
Abstract (arXiv)
Graph Neural Networks (GNNs) are non-Euclidean deep learning models for graph-structured data. Despite their successful and diverse applications, oversmoothing prohibits deep architectures due to node features converging to a single fixed point. This severely limits their potential to solve complex tasks. To counteract this tendency, we propose a plug-and-play module consisting of three steps: Cluster-Normalize-Activate (CNA). By applying CNA modules, GNNs search and form super nodes in each layer, which are normalized and activated individually. We demonstrate in node classification and property prediction tasks that CNA significantly improves the accuracy over the state-of-the-art. Particularly, CNA reaches 94.18% and 95.75% accuracy on Cora and CiteSeer, respectively. It further benefits GNNs in regression tasks as well, reducing the mean squared error compared to all baselines. At the same time, GNNs with CNA require substantially fewer learnable parameters than competing architectures.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 54 of its 78 references that are in the map. First posted on arXiv 5 Dec 2024.
Builds on
54 of 78 references.
References · 54
- 2017Attention Is All You Needcited by 59,932
- 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
- 2017Graph Attention Networkscited by 8,854
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
- 2017Inductive Representation Learning on Large Graphscited by 6,284
- 2017Neural Message Passing for Quantum Chemistrycited by 3,460
- 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,414
- 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
- 2018Graph Neural Networks: A Review of Methods and Applicationscited by 1,571
- 2019Simplifying Graph Convolutional Networkscited by 1,470
- 2018Relational inductive biases, deep learning, and graph networkscited by 1,341
- 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
38 more
- 2018Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learningcited by 1,099
- 2018Pitfalls of Graph Neural Network Evaluationcited by 994
- 2018Link Prediction Based on Graph Neural Networkscited by 829
- 2020Geom-GCN: Geometric Graph Convolutional Networkscited by 821
- 2020TUDataset: A collection of benchmark datasets for learning with graphscited by 718
- 2020On the Bottleneck of Graph Neural Networks and its Practical Implicationscited by 624
- 2019Multi-scale Attributed Node Embeddingcited by 543
- 2019Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networkscited by 540
- 2020Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classificationcited by 500
- 2021Understanding over-squashing and bottlenecks on graphs via curvaturecited by 471
- 2017Deep Gaussian Embedding of Graphs: Unsupervised Inductive Learning via Rankingcited by 332
- 2019PairNorm: Tackling Oversmoothing in GNNscited by 297
- 2019Revisiting Graph Neural Networks: All We Have is Low-Pass Filterscited by 260
- 2021Graph Neural Network for Traffic Forecasting: A Surveycited by 260
- 2023A Survey on Oversmoothing in Graph Neural Networkscited by 246
- 2021Training Graph Neural Networks with 1000 Layerscited by 169
- 2022Revisiting Heterophily For Graph Neural Networkscited by 169
- 2023Exphormer: Sparse Transformers for Graphscited by 166
- 2023Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learningcited by 132
- 2022Graph-Coupled Oscillator Networkscited by 123
- 2020GraphNorm: A Principled Approach to Accelerating Graph Neural Network Trainingcited by 116
- 2020Towards Deeper Graph Neural Networks with Differentiable Group Normalizationcited by 113
- 2022Revisiting Over-smoothing and Over-squashing Using Ollivier-Ricci Curvaturecited by 96
- 2020Normalization Techniques in Training DNNs: Methodology, Analysis and Applicationcited by 90
- 2020A survey on modern trainable activation functionscited by 89
- 2023Edge Directionality Improves Learning on Heterophilic Graphscited by 88
- 2020Rational neural networkscited by 75
- 2023SimTeG: A Frustratingly Simple Approach Improves Textual Graph Learningcited by 73
- 2022Revisiting Over-smoothing in BERT from the Perspective of Graphcited by 71
- 2019Pad'e Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networkscited by 65
- 2022Gradient Gating for Deep Multi-Rate Learning on Graphscited by 49
- 2017Neural networks and rational functionscited by 40
- 2020Optimization of Graph Neural Networks with Natural Gradient Descentcited by 26
- 2021Adaptive Rational Activations to Boost Deep Reinforcement Learningcited by 23
- 2022Graph Neural Networks for Moleculescited by 21
- 2020Learning Graph Normalization for Graph Neural Networkscited by 16
- 2023Higher-order Graph Convolutional Network with Flower-Petals Laplacians on Simplicial Complexescited by 15
- 2021Distilling Self-Knowledge From Contrastive Links to Classify Graph Nodes Without Passing Messagescited by 11
50 most cited of 54 All 54 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.