# Graph-Aware Transformer: Is Attention All Graphs Need?

arXiv: https://arxiv.org/abs/2006.05213 · HTML: https://mlsanity.com/paper/2006.05213/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jun 2020.

## Built on by (most cited first, 6 of 6)

- [XAI for Transformers: Better Explanations through Conservative Propagation](https://mlsanity.com/paper/2202.07304/) (2022)
- [Permutation invariant graph-to-sequence model for template-free retrosynthesis and reaction prediction](https://mlsanity.com/paper/2110.09681/) (2021)
- [The Future is not One-dimensional: Complex Event Schema Induction by Graph Modeling for Event Prediction](https://mlsanity.com/paper/2104.06344/) (2021)
- [Image-to-Graph Transformers for Chemical Structure Recognition](https://mlsanity.com/paper/2202.09580/) (2022)
- [Pre-training Transformers for Molecular Property Prediction Using Reaction Prediction](https://mlsanity.com/paper/2207.02724/) (2022)
- [GeoT: A Geometry-aware Transformer for Reliable Molecular Property Prediction and Chemically Interpretable Representation Learning](https://mlsanity.com/paper/2106.15516/) (2021)

## Abstract (arXiv)

Graphs are the natural data structure to represent relational and structural information in many domains. To cover the broad range of graph-data applications including graph classification as well as graph generation, it is desirable to have a general and flexible model consisting of an encoder and a decoder that can handle graph data. Although the representative encoder-decoder model, Transformer, shows superior performance in various tasks especially of natural language processing, it is not immediately available for graphs due to their non-sequential characteristics. To tackle this incompatibility, we propose GRaph-Aware Transformer (GRAT), the first Transformer-based model which can encode and decode whole graphs in end-to-end fashion. GRAT is featured with a self-attention mechanism adaptive to the edge information and an auto-regressive decoding mechanism based on the two-path approach consisting of sub-graph encoding path and node-and-edge generation path for each decoding step. We empirically evaluated GRAT on multiple setups including encoder-based tasks such as molecule property predictions on QM9 datasets and encoder-decoder-based tasks such as molecule graph generation in the organic molecule synthesis domain. GRAT has shown very promising results including state-of-the-art performance on 4 regression tasks in QM9 benchmark.