mlsanityData to 13 Oct 2026

Relieving the Over-Aggregating Effect in Graph Transformers

arXiv 2510.21267 ↗First posted 24 Oct 2025cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Graph attention has demonstrated superior performance in graph learning tasks. However, learning from global interactions can be challenging due to the large number of nodes. In this paper, we discover a new phenomenon termed over-aggregating. Over-aggregating arises when a large volume of messages is aggregated into a single node with less discrimination, leading to the dilution of the key messages and potential information loss. To address this, we propose Wideformer, a plug-and-play method for graph attention. Wideformer divides the aggregation of all nodes into parallel processes and guides the model to focus on specific subsets of these processes. The division can limit the input volume per aggregation, avoiding message dilution and reducing information loss. The guiding step sorts and weights the aggregation outputs, prioritizing the informative messages. Evaluations show that Wideformer can effectively mitigate over-aggregating. As a result, the backbone methods can focus on the informative messages, achieving superior performance compared to baseline methods.

Key facts. Cited by 2 papers in the map (data through 13 Oct 2026). Builds on 46 of its 63 references that are in the map. First posted on arXiv 24 Oct 2025.

Builds on

46 of 63 references.

References · 46

  1. 2017Attention Is All You Needcited by 60,106
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,223
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,111
  4. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,318
  5. 2017Graph Attention Networkscited by 8,888
  6. 2017Inductive Representation Learning on Large Graphscited by 6,303
  7. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,422
  8. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,157
  9. 2021ViViT: A Video Vision Transformercited by 1,615
  10. 2020Rethinking Attention with Performerscited by 1,594
  11. 2018Pitfalls of Graph Neural Network Evaluationcited by 1,003
  12. 2018Hierarchical Graph Representation Learning with Differentiable Poolingcited by 787
34 more
  1. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 747
  2. 2022Recipe for a General, Powerful, Scalable Graph Transformercited by 709
  3. 2020A Generalization of Transformer Networks to Graphscited by 658
  4. 2020On the Bottleneck of Graph Neural Networks and its Practical Implicationscited by 629
  5. 2020Adaptive Universal Generalized PageRank Graph Neural Networkcited by 595
  6. 2019Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networkscited by 541
  7. 2019Graph Transformer Networkscited by 522
  8. 2021Understanding over-squashing and bottlenecks on graphs via curvaturecited by 473
  9. 2021Rethinking Graph Transformers with Spectral Attentioncited by 472
  10. 2019Self-Attention Graph Poolingcited by 421
  11. 2023A critical look at the evaluation of GNNs under heterophily: Are we really making progress?cited by 337
  12. 2021Graph Neural Networks with Learnable Structural and Positional Representationscited by 331
  13. 2021Do Transformers Really Perform Bad for Graph Representation?cited by 319
  14. 2021Large Scale Learning on Non-Homophilous Graphs: New Benchmarks and Strong Simple Methodscited by 311
  15. 2023NodeFormer: A Scalable Graph Structure Learning Transformer for Node Classificationcited by 209
  16. 2022Representing Long-Range Context for Graph Neural Networks with Global Attentioncited by 193
  17. 2020Graph-Bert: Only Attention is Needed for Learning Graph Representationscited by 177
  18. 2023Exphormer: Sparse Transformers for Graphscited by 169
  19. 2021GraphiT: Encoding Graph Structure in Transformerscited by 147
  20. 2022NAGphormer: A Tokenized Graph Transformer for Node Classification in Large Graphscited by 137
  21. 2023SGFormer: Simplifying and Empowering Transformers for Large-Graph Representationscited by 132
  22. 2022Pre-training via Denoising for Molecular Property Predictioncited by 124
  23. 2023DIFFormer: Scalable (Graph) Transformers Induced by Energy Constrained Diffusioncited by 86
  24. 2023DRew: Dynamically Rewired Message Passing with Delaycited by 82
  25. 2020Rethinking pooling in graph neural networkscited by 73
  26. 2024Polynormer: Polynomial-Expressive Graph Transformer in Linear Timecited by 65
  27. 2023A Complete Expressiveness Hierarchy for Subgraph GNNs via Subgraph Weisfeiler-Lehman Testscited by 64
  28. 2022Scratching Visual Transformer's Back with Uniform Attentioncited by 16
  29. 2024Even Sparser Graph Transformerscited by 13
  30. 2024Towards Dynamic Message Passing on Graphscited by 11
  31. 2024Expanding Sparse Tuning for Low Memory Usagecited by 10
  32. 2024Scalable Graph Compressed Convolutionscited by 2
  33. 2024ENACT: Entropy-based Clustering of Attention Input for Reducing the Computational Needs of Object Detection Transformerscited by 1
  34. 2025Understanding and Tackling Over-Dilution in Graph Neural Networkscited by 1

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Edit Less, Achieve More: Dynamic Sparse Neuron Masking for Lifelong Knowledge Editing in LLMscited by 7
  2. 2026Distance-Misaligned Training in Graph Transformers and Adaptive Graph-Aware Controlcited by 0