# Graph-adaptive Rectified Linear Unit for Graph Neural Networks

arXiv: https://arxiv.org/abs/2202.06281 · HTML: https://mlsanity.com/paper/2202.06281/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 48 references that are in the map.
- First posted on arXiv 13 Feb 2022.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering](https://mlsanity.com/paper/1606.09375/) (2016)
- [Simplifying Graph Convolutional Networks](https://mlsanity.com/paper/1902.07153/) (2019)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Predict then Propagate: Graph Neural Networks meet Personalized PageRank](https://mlsanity.com/paper/1810.05997/) (2018)
- [Gated Graph Sequence Neural Networks](https://mlsanity.com/paper/1511.05493/) (2015)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [Empirical Evaluation of Rectified Activations in Convolutional Network](https://mlsanity.com/paper/1505.00853/) (2015)
- [MAGNN: Metapath Aggregated Graph Neural Network for Heterogeneous Graph Embedding](https://mlsanity.com/paper/2002.01680/) (2020)
- [Maxout Networks](https://mlsanity.com/paper/1302.4389/) (2013)
- [Graph Neural Networks with convolutional ARMA filters](https://mlsanity.com/paper/1901.01343/) (2019)
- [Mish: A Self Regularized Non-Monotonic Activation Function](https://mlsanity.com/paper/1908.08681/) (2019)
- [Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units](https://mlsanity.com/paper/1603.05201/) (2016)
- [Graph-based Semi-supervised Learning: A Comprehensive Review](https://mlsanity.com/paper/2102.13303/) (2021)
- [Discrete-time Temporal Network Embedding via Implicit Hierarchical Learning in Hyperbolic Space](https://mlsanity.com/paper/2107.03767/) (2021)
- [Dynamic ReLU](https://mlsanity.com/paper/2003.10027/) (2020)
- [Convolutional Kernel Networks](https://mlsanity.com/paper/1406.3332/) (2014)
- [Modeling Scale-free Graphs with Hyperbolic Geometry for Knowledge-aware Recommendation](https://mlsanity.com/paper/2108.06468/) (2021)
- [STP-UDGAT: Spatial-Temporal-Preference User Dimensional Graph Attention Network for Next POI Recommendation](https://mlsanity.com/paper/2010.07024/) (2020)
- [Power Normalizations in Fine-grained Image, Few-shot Image and Graph Classification](https://mlsanity.com/paper/2012.13975/) (2020)
- [Contrastive Laplacian Eigenmaps](https://mlsanity.com/paper/2201.05493/) (2022)
- [FeatureNorm: L2 Feature Normalization for Dynamic Graph Embedding](https://mlsanity.com/paper/2103.00164/) (2021)
- [Fisher-Bures Adversary Graph Convolutional Networks](https://mlsanity.com/paper/1903.04154/) (2019)
- [A Deeper Look at Power Normalizations](https://mlsanity.com/paper/1806.09183/) (2018)

## Built on by (most cited first, 12 of 12)

- [Spectral Feature Augmentation for Graph Contrastive Learning and Beyond](https://mlsanity.com/paper/2212.01026/) (2022)
- [COSTA: Covariance-Preserving Feature Augmentation for Graph Contrastive Learning](https://mlsanity.com/paper/2206.04726/) (2022)
- [HICF: Hyperbolic Informative Collaborative Filtering](https://mlsanity.com/paper/2207.09051/) (2022)
- [A Survey of Trustworthy Federated Learning with Perspectives on Security, Robustness, and Privacy](https://mlsanity.com/paper/2302.10637/) (2023)
- [Uncertainty-DTW for Time Series and Sequences](https://mlsanity.com/paper/2211.00005/) (2022)
- [Because Every Sensor Is Unique, so Is Every Pair: Handling Dynamicity in Traffic Forecasting](https://mlsanity.com/paper/2302.09956/) (2023)
- [Graph Component Contrastive Learning for Concept Relatedness Estimation](https://mlsanity.com/paper/2206.12556/) (2022)
- [Exploring Heterogeneity and Uncertainty for Graph-based Cognitive Diagnosis Models in Intelligent Education](https://mlsanity.com/paper/2403.05559/) (2024)
- [DiGRAF: Diffeomorphic Graph-Adaptive Activation Function](https://mlsanity.com/paper/2407.02013/) (2024)
- [FedHGN: A Federated Framework for Heterogeneous Graph Neural Networks](https://mlsanity.com/paper/2305.09729/) (2023)
- [Degree-based stratification of nodes in Graph Neural Networks](https://mlsanity.com/paper/2312.10458/) (2023)
- [Graph Self-Supervised Learning with Learnable Structural and Positional Encodings](https://mlsanity.com/paper/2502.16233/) (2025)

## Abstract (arXiv)

Graph Neural Networks (GNNs) have achieved remarkable success by extending traditional convolution to learning on non-Euclidean data. The key to the GNNs is adopting the neural message-passing paradigm with two stages: aggregation and update. The current design of GNNs considers the topology information in the aggregation stage. However, in the updating stage, all nodes share the same updating function. The identical updating function treats each node embedding as i.i.d. random variables and thus ignores the implicit relationships between neighborhoods, which limits the capacity of the GNNs. The updating function is usually implemented with a linear transformation followed by a non-linear activation function. To make the updating function topology-aware, we inject the topological information into the non-linear activation function and propose Graph-adaptive Rectified Linear Unit (GReLU), which is a new parametric activation function incorporating the neighborhood information in a novel and efficient way. The parameters of GReLU are obtained from a hyperfunction based on both node features and the corresponding adjacent matrix. To reduce the risk of overfitting and the computational cost, we decompose the hyperfunction as two independent components for nodes and features respectively. We conduct comprehensive experiments to show that our plug-and-play GReLU method is efficient and effective given different GNN backbones and various downstream tasks.