mlsanityData to 14 Oct 2026

Self-Attention in Colors: Another Take on Encoding Graph Structure in Transformers

arXiv 2304.10933 ↗First posted 21 Apr 2023cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

We introduce a novel self-attention mechanism, which we call CSA (Chromatic Self-Attention), which extends the notion of attention scores to attention _filters_, independently modulating the feature channels. We showcase CSA in a fully-attentional graph Transformer CGT (Chromatic Graph Transformer) which integrates both graph structural information and edge features, completely bypassing the need for local message-passing components. Our method flexibly encodes graph structure through node-node interactions, by enriching the original edge features with a relative positional encoding scheme. We propose a new scheme based on random walks that encodes both structural and positional information, and show how to incorporate higher-order topological information, such as rings in molecular graphs. Our approach achieves state-of-the-art results on the ZINC benchmark dataset, while providing a flexible framework for encoding graph structure and incorporating higher-order topology.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 35 of its 42 references that are in the map. First posted on arXiv 21 Apr 2023.

Builds on

35 of 42 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  3. 2017Graph Attention Networkscited by 8,923
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  5. 2018How Powerful are Graph Neural Networks?cited by 4,393
  6. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,163
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  8. 2018Graph Convolutional Neural Networks for Web-Scale Recommender Systemscited by 1,138
  9. 2018Self-Attention with Relative Position Representationscited by 1,120
  10. 2019Strategies for Pre-training Graph Neural Networkscited by 971
  11. 2022Recipe for a General, Powerful, Scalable Graph Transformercited by 713
  12. 2020Benchmarking Graph Neural Networkscited by 680
23 more
  1. 2020On the Bottleneck of Graph Neural Networks and its Practical Implicationscited by 631
  2. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 629
  3. 2020Principal Neighbourhood Aggregation for Graph Netscited by 478
  4. 2021Rethinking Graph Transformers with Spectral Attentioncited by 473
  5. 2017Residual Gated Graph ConvNetscited by 346
  6. 2021Graph Neural Networks with Learnable Structural and Positional Representationscited by 333
  7. 2021OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphscited by 329
  8. 2020Improving Graph Neural Network Expressivity via Subgraph Isomorphism Countingcited by 313
  9. 2022Long Range Graph Benchmarkcited by 263
  10. 2021Weisfeiler and Lehman Go Topological: Message Passing Simplicial Networkscited by 221
  11. 2021Weisfeiler and Lehman Go Cellular: CW Networkscited by 218
  12. 2020The Surprising Power of Graph Neural Networks with Random Node Initializationcited by 198
  13. 2022Representing Long-Range Context for Graph Neural Networks with Global Attentioncited by 195
  14. 2022Structure-Aware Transformer for Graph Representation Learningcited by 194
  15. 2022Pure Transformers are Powerful Graph Learnerscited by 188
  16. 2022Sign and Basis Invariant Networks for Spectral Graph Representation Learningcited by 179
  17. 2021From Stars to Subgraphs: Uplifting Any GNN with Local Structure Awarenesscited by 155
  18. 2021GraphiT: Encoding Graph Structure in Transformerscited by 147
  19. 2021ETA Prediction with Graph Neural Networks in Google Mapscited by 130
  20. 2021Weisfeiler and Leman go Machine Learning: The Story so farcited by 122
  21. 2021Global Self-Attention as a Replacement for Graph Convolutioncited by 114
  22. 2022GRPE: Relative Positional Encoding for Graph Transformercited by 70
  23. 2020Hierarchical Inter-Message Passing for Learning on Molecular Graphscited by 59

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 4; 2025: 2.

7 papers.

  1. 2024An end-to-end attention-based approach for learning on graphscited by 11
  2. 2024Graph Transformers without Positional Encodingscited by 2
  3. 2025Simple Path Structural Encoding for Graph Transformerscited by 1
  4. 2023Hybrid Focal and Full-Range Attention Based Graph Transformerscited by 0
  5. 2024Differential Encoding for Improved Representation Learning over Graphscited by 0
  6. 2024Range-aware Positional Encoding via High-order Pretraining: Theory and Practicecited by 0
  7. 2025Challenges in Non-Polymeric Crystal Structure Prediction: Why a Geometric, Permutation-Invariant Loss is Neededcited by 0