mlsanityData to 7 Oct 2026

Hybrid Focal and Full-Range Attention Based Graph Transformers

arXiv 2311.04653 ↗First posted 8 Nov 2023cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

The paradigm of Transformers using the self-attention mechanism has manifested its advantage in learning graph-structured data. Yet, Graph Transformers are capable of modeling full range dependencies but are often deficient in extracting information from locality. A common practice is to utilize Message Passing Neural Networks (MPNNs) as an auxiliary to capture local information, which however are still inadequate for comprehending substructures. In this paper, we present a purely attention-based architecture, namely Focal and Full-Range Graph Transformer (FFGT), which can mitigate the loss of local information in learning global correlations. The core component of FFGT is a new mechanism of compound attention, which combines the conventional full-range attention with K-hop focal attention on ego-nets to aggregate both global and local information. Beyond the scope of canonical Transformers, the FFGT has the merit of being more substructure-aware. Our approach enhances the performance of existing Graph Transformers on various open datasets, while achieves compatible SOTA performance on several Long-Range Graph Benchmark (LRGB) datasets even with a vanilla transformer. We further examine influential factors on the optimal focal length of attention via introducing a novel synthetic dataset based on SBM-PATTERN.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 42 of its 50 references that are in the map. First posted on arXiv 8 Nov 2023.

Builds on

42 of 50 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  5. 2017Graph Attention Networkscited by 8,854
  6. 2018How Powerful are Graph Neural Networks?cited by 4,355
  7. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
  8. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
  9. 2018Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learningcited by 1,099
  10. 2020Geom-GCN: Geometric Graph Convolutional Networkscited by 821
  11. 2022Recipe for a General, Powerful, Scalable Graph Transformercited by 702
  12. 2020Benchmarking Graph Neural Networkscited by 672
30 more
  1. 2020A Generalization of Transformer Networks to Graphscited by 653
  2. 2020On the Bottleneck of Graph Neural Networks and its Practical Implicationscited by 624
  3. 2019Multi-scale Attributed Node Embeddingcited by 543
  4. 2020Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classificationcited by 500
  5. 2021Understanding over-squashing and bottlenecks on graphs via curvaturecited by 471
  6. 2021Rethinking Graph Transformers with Spectral Attentioncited by 468
  7. 2017Residual Gated Graph ConvNetscited by 341
  8. 2021Mesh Graphormercited by 264
  9. 2022Long Range Graph Benchmarkcited by 258
  10. 2022Representing Long-Range Context for Graph Neural Networks with Global Attentioncited by 193
  11. 2022Pure Transformers are Powerful Graph Learnerscited by 188
  12. 2020Graph-Bert: Only Attention is Needed for Learning Graph Representationscited by 176
  13. 2023Exphormer: Sparse Transformers for Graphscited by 166
  14. 2023Graph Inductive Biases in Transformers without Message Passingcited by 150
  15. 2021GraphiT: Encoding Graph Structure in Transformerscited by 146
  16. 2022NAGphormer: A Tokenized Graph Transformer for Node Classification in Large Graphscited by 137
  17. 2021Global Self-Attention as a Replacement for Graph Convolutioncited by 113
  18. 2022Understanding and Extending Subgraph GNNs by Rethinking Their Symmetriescited by 112
  19. 2022Transformer for Graphs: An Overview from Architecture Perspectivecited by 107
  20. 2022A Generalization of ViT/MLP-Mixer to Graphscited by 93
  21. 2022How Powerful are K-hop Message Passing Graph Neural Networkscited by 90
  22. 2023Attending to Graph Transformerscited by 81
  23. 2020Subgraph Neural Networkscited by 73
  24. 2022GRPE: Relative Positional Encoding for Graph Transformercited by 67
  25. 2021Gophormer: Ego-Graph Transformer for Node Classificationcited by 63
  26. 2019k-hop Graph Neural Networkscited by 57
  27. 2022Your Transformer May Not be as Powerful as You Expectcited by 55
  28. 2023Graph Propagation Transformer for Graph Representation Learningcited by 9
  29. 2023Self-Attention in Colors: Another Take on Encoding Graph Structure in Transformerscited by 7
  30. 2023Multiresolution Graph Transformers and Wavelet Positional Encoding for Learning Hierarchical Structurescited by 4

Built on by

Not cited yet.