# Improving Graph Attention Networks with Large Margin-based Constraints

arXiv: https://arxiv.org/abs/1910.11945 · HTML: https://mlsanity.com/paper/1910.11945/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Oct 2019.

## Built on by (most cited first, 30 of 34)

- [How Attentive are Graph Attention Networks?](https://mlsanity.com/paper/2105.14491/) (2021)
- [Large-Scale Representation Learning on Graphs via Bootstrapping](https://mlsanity.com/paper/2102.06514/) (2021)
- [Two Sides of the Same Coin: Heterophily and Oversmoothing in Graph Convolutional Neural Networks](https://mlsanity.com/paper/2102.06462/) (2021)
- [How to Find Your Friendly Neighborhood: Graph Attention Design with Self-Supervision](https://mlsanity.com/paper/2204.04879/) (2022)
- [Demystifying Oversmoothing in Attention-Based Graph Neural Networks](https://mlsanity.com/paper/2305.16102/) (2023)
- [Graph Neural Networks with Learnable and Optimal Polynomial Bases](https://mlsanity.com/paper/2302.12432/) (2023)
- [Learning Conjoint Attentions for Graph Neural Nets](https://mlsanity.com/paper/2102.03147/) (2021)
- [Towards Deep Attention in Graph Neural Networks: Problems and Remedies](https://mlsanity.com/paper/2306.02376/) (2023)
- [Improving Graph Neural Networks with Learnable Propagation Operators](https://mlsanity.com/paper/2210.17224/) (2022)
- [Adaptive Kernel Graph Neural Network](https://mlsanity.com/paper/2112.04575/) (2021)
- [Exploiting Neighbor Effect: Conv-Agnostic GNNs Framework for Graphs with Heterophily](https://mlsanity.com/paper/2203.11200/) (2022)
- [SkipNode: On Alleviating Performance Degradation for Deep Graph Convolutional Networks](https://mlsanity.com/paper/2112.11628/) (2021)
- [Beyond Over-smoothing: Uncovering the Trainability Challenges in Deep Graph Neural Networks](https://mlsanity.com/paper/2408.03669/) (2024)
- [Graph Decoupling Attention Markov Networks for Semi-supervised Graph Node Classification](https://mlsanity.com/paper/2104.13718/) (2021)
- [Unleashing the Power of Transformer for Graphs](https://mlsanity.com/paper/2202.10581/) (2022)
- [From Cluster Assumption to Graph Convolution: Graph-based Semi-Supervised Learning Revisited](https://mlsanity.com/paper/2309.13599/) (2023)
- [Clenshaw Graph Neural Networks](https://mlsanity.com/paper/2210.16508/) (2022)
- [Causal-Based Supervision of Attention in Graph Neural Network: A Better and Simpler Choice towards Powerful Attention](https://mlsanity.com/paper/2305.13115/) (2023)
- [Tailoring Self-Attention for Graph via Rooted Subtrees](https://mlsanity.com/paper/2310.05296/) (2023)
- [GATE: How to Keep Out Intrusive Neighbors](https://mlsanity.com/paper/2406.00418/) (2024)
- [MaxCutPool: differentiable feature-aware Maxcut for pooling in graph neural networks](https://mlsanity.com/paper/2409.05100/) (2024)
- [Deepened Graph Auto-Encoders Help Stabilize and Enhance Link Prediction](https://mlsanity.com/paper/2103.11414/) (2021)
- [When Graph Neural Network Meets Causality: Opportunities, Methodologies and An Outlook](https://mlsanity.com/paper/2312.12477/) (2023)
- [GTNet: A Tree-Based Deep Graph Learning Architecture](https://mlsanity.com/paper/2204.12802/) (2022)
- [Comprehensive Analysis of Over-smoothing in Graph Neural Networks from Markov Chains Perspective](https://mlsanity.com/paper/2211.06605/) (2022)
- [Triplet Edge Attention for Algorithmic Reasoning](https://mlsanity.com/paper/2312.05611/) (2023)
- [DuetGraph: Coarse-to-Fine Knowledge Graph Reasoning with Dual-Pathway Global-Local Fusion](https://mlsanity.com/paper/2507.11229/) (2025)
- [Adaptive Initial Residual Connections for GNNs with Theoretical Guarantees](https://mlsanity.com/paper/2511.06598/) (2025)
- [KAA: Kolmogorov-Arnold Attention for Enhancing Attentive Graph Neural Networks](https://mlsanity.com/paper/2501.13456/) (2025)
- [Urban Incident Prediction with Graph Neural Networks: Integrating Government Ratings and Crowdsourced Reports](https://mlsanity.com/paper/2506.08740/) (2025)

## Abstract (arXiv)

Graph Attention Networks (GATs) are the state-of-the-art neural architecture for representation learning with graphs. GATs learn attention functions that assign weights to nodes so that different nodes have different influences in the feature aggregation steps. In practice, however, induced attention functions are prone to over-fitting due to the increasing number of parameters and the lack of direct supervision on attention weights. GATs also suffer from over-smoothing at the decision boundary of nodes. Here we propose a framework to address their weaknesses via margin-based constraints on attention during training. We first theoretically demonstrate the over-smoothing behavior of GATs and then develop an approach using constraint on the attention weights according to the class boundary and feature aggregation pattern. Furthermore, to alleviate the over-fitting problem, we propose additional constraints on the graph structure. Extensive experiments and ablation studies on common benchmark datasets demonstrate the effectiveness of our method, which leads to significant improvements over the previous state-of-the-art graph attention methods on all datasets.