mlsanityData to 7 Oct 2026

Diffusing Graph Attention

arXiv 2303.00613 ↗First posted 1 Mar 2023cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

The dominant paradigm for machine learning on graphs uses Message Passing Graph Neural Networks (MP-GNNs), in which node representations are updated by aggregating information in their local neighborhood. Recently, there have been increasingly more attempts to adapt the Transformer architecture to graphs in an effort to solve some known limitations of MP-GNN. A challenging aspect of designing Graph Transformers is integrating the arbitrary graph structure into the architecture. We propose Graph Diffuser (GD) to address this challenge. GD learns to extract structural and positional relationships between distant nodes in the graph, which it then uses to direct the Transformer's attention and node representation. We demonstrate that existing GNNs and Graph Transformers struggle to capture long-range interactions and how Graph Diffuser does so while admitting intuitive visualizations. Experiments on eight benchmarks show Graph Diffuser to be a highly competitive model, outperforming the state-of-the-art in a diverse set of domains.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 54 of its 65 references that are in the map. First posted on arXiv 1 Mar 2023.

Builds on

54 of 65 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  3. 2017Graph Attention Networkscited by 8,854
  4. 2020End-to-End Object Detection with Transformerscited by 7,384
  5. 2017Inductive Representation Learning on Large Graphscited by 6,284
  6. 2018How Powerful are Graph Neural Networks?cited by 4,355
  7. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
  8. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,906
  9. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,414
  10. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
  11. 2019Simplifying Graph Convolutional Networkscited by 1,470
  12. 2018Self-Attention with Relative Position Representationscited by 1,112
38 more
  1. 2018Predict then Propagate: Graph Neural Networks meet Personalized PageRankcited by 987
  2. 2018Representation Learning on Graphs with Jumping Knowledge Networkscited by 875
  3. 2021How Attentive are Graph Attention Networks?cited by 818
  4. 2015Gated Graph Sequence Neural Networkscited by 792
  5. 2019Heterogeneous Graph Attention Networkcited by 732
  6. 2022Recipe for a General, Powerful, Scalable Graph Transformercited by 702
  7. 2020Benchmarking Graph Neural Networkscited by 672
  8. 2020A Generalization of Transformer Networks to Graphscited by 653
  9. 2020A Survey on Visual Transformercited by 653
  10. 2020On the Bottleneck of Graph Neural Networks and its Practical Implicationscited by 624
  11. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  12. 2020Principal Neighbourhood Aggregation for Graph Netscited by 477
  13. 2021Understanding over-squashing and bottlenecks on graphs via curvaturecited by 471
  14. 2021Rethinking Graph Transformers with Spectral Attentioncited by 468
  15. 2019Diffusion Improves Graph Learningcited by 414
  16. 2017Residual Gated Graph ConvNetscited by 341
  17. 2021OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphscited by 328
  18. 2021Graph Neural Networks with Learnable Structural and Positional Representationscited by 326
  19. 2021Do Transformers Really Perform Bad for Graph Representation?cited by 318
  20. 2021CMT: Convolutional Neural Networks Meet Vision Transformerscited by 310
  21. 2015Diffusion-Convolutional Neural Networkscited by 268
  22. 2022Long Range Graph Benchmarkcited by 258
  23. 2020DeeperGCN: All You Need to Train Deeper GCNscited by 249
  24. 2019Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networkscited by 228
  25. 2022Representing Long-Range Context for Graph Neural Networks with Global Attentioncited by 193
  26. 2022Structure-Aware Transformer for Graph Representation Learningcited by 191
  27. 2020Design Space for Graph Neural Networkscited by 182
  28. 2021Thinking Like Transformerscited by 182
  29. 2022Sign and Basis Invariant Networks for Spectral Graph Representation Learningcited by 176
  30. 2021From Stars to Subgraphs: Uplifting Any GNN with Local Structure Awarenesscited by 154
  31. 2021GraphiT: Encoding Graph Structure in Transformerscited by 146
  32. 2020Directional Graph Networkscited by 132
  33. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  34. 2020Distilling Knowledge from Graph Convolutional Networkscited by 128
  35. 2019Learning Attention-based Embeddings for Relation Prediction in Knowledge Graphscited by 125
  36. 2022Equivariant and Stable Positional Encoding for More Powerful Graph Neural Networkscited by 94
  37. 2020My Body is a Cage: the Role of Morphology in Graph-Based Incompatible Controlcited by 68
  38. 2022GRPE: Relative Positional Encoding for Graph Transformercited by 67

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 4; 2025: 1.

5 papers.

  1. 2024Towards Principled Graph Transformerscited by 11
  2. 2024Aligning Transformers with Weisfeiler-Lemancited by 8
  3. 2025Can You Hear Me Now? A Benchmark for Long-Range Graph Propagationcited by 8
  4. 2024Scalable Graph Compressed Convolutionscited by 2
  5. 2024Differential Encoding for Improved Representation Learning over Graphscited by 0