# Improving Attention Mechanism in Graph Neural Networks via Cardinality Preservation

arXiv: https://arxiv.org/abs/1907.02204 · HTML: https://mlsanity.com/paper/1907.02204/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jul 2019.

## Built on by (most cited first, 17 of 17)

- [A Survey on Graph Representation Learning Methods](https://mlsanity.com/paper/2204.01855/) (2022)
- [Learning Conjoint Attentions for Graph Neural Nets](https://mlsanity.com/paper/2102.03147/) (2021)
- [Graph Ordering Attention Networks](https://mlsanity.com/paper/2204.05351/) (2022)
- [A Survey on Graph Classification and Link Prediction based on GNN](https://mlsanity.com/paper/2307.00865/) (2023)
- [Not All Neighbors Are Worth Attending to: Graph Selective Attention Networks for Semi-supervised Learning](https://mlsanity.com/paper/2210.07715/) (2022)
- [AGFormer: Efficient Graph Representation with Anchor-Graph Transformer](https://mlsanity.com/paper/2305.07521/) (2023)
- [Controversy Detection: a Text and Graph Neural Network Based Approach](https://mlsanity.com/paper/2112.11445/) (2021)
- [Attention-based graph neural networks: a survey](https://mlsanity.com/paper/2605.08679/) (2026)
- [Mergeable Model-Side Aggregation States for Long-Context Language Models](https://mlsanity.com/paper/2607.26448/) (2026)
- [Quadratic GCN for Graph Classification](https://mlsanity.com/paper/2104.06750/) (2021)
- [Unified GCNs: Towards Connecting GCNs with CNNs](https://mlsanity.com/paper/2204.12300/) (2022)
- [CardiGraphormer: Unveiling the Power of Self-Supervised Learning in Revolutionizing Drug Discovery](https://mlsanity.com/paper/2307.00859/) (2023)
- [Can we ease the Injectivity Bottleneck on Lorentzian Manifolds for Graph Neural Networks?](https://mlsanity.com/paper/2504.00142/) (2025)
- [Optimal message passing for molecular prediction is simple, attentive and spatial](https://mlsanity.com/paper/2509.10871/) (2025)
- [Cardinality-Preserving Attention Channels for Graph Transformers in Molecular Property Prediction](https://mlsanity.com/paper/2602.02201/) (2026)
- [What Softmax Throws Away: Mass-Aware Attention for Evidence Accumulation](https://mlsanity.com/paper/2607.22781/) (2026)
- [Escaping Oversquashing: Addressable and Support-Aware Global Memory for Message Passing Networks](https://mlsanity.com/paper/2608.02709/) (2026)

## Abstract (arXiv)

Graph Neural Networks (GNNs) are powerful to learn the representation of graph-structured data. Most of the GNNs use the message-passing scheme, where the embedding of a node is iteratively updated by aggregating the information of its neighbors. To achieve a better expressive capability of node influences, attention mechanism has grown to be popular to assign trainable weights to the nodes in aggregation. Though the attention-based GNNs have achieved remarkable results in various tasks, a clear understanding of their discriminative capacities is missing. In this work, we present a theoretical analysis of the representational properties of the GNN that adopts the attention mechanism as an aggregator. Our analysis determines all cases when those attention-based GNNs can always fail to distinguish certain distinct structures. Those cases appear due to the ignorance of cardinality information in attention-based aggregation. To improve the performance of attention-based GNNs, we propose cardinality preserved attention (CPA) models that can be applied to any kind of attention mechanisms. Our experiments on node and graph classification confirm our theoretical analysis and show the competitive performance of our CPA models.