# Learning Attributed Graph Representations with Communicative Message Passing Transformer

arXiv: https://arxiv.org/abs/2107.08773 · HTML: https://mlsanity.com/paper/2107.08773/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 28 references that are in the map.
- First posted on arXiv 19 Jul 2021.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning](https://mlsanity.com/paper/1801.07606/) (2018)
- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Convolutional Networks on Graphs for Learning Molecular Fingerprints](https://mlsanity.com/paper/1509.09292/) (2015)
- [Molecular Graph Convolutions: Moving Beyond Fingerprints](https://mlsanity.com/paper/1603.00856/) (2016)
- [N-Gram Graph: Simple Unsupervised Representation for Graphs, with Applications to Molecules](https://mlsanity.com/paper/1806.09206/) (2018)
- [Molecule Attention Transformer](https://mlsanity.com/paper/2002.08264/) (2020)
- [SMILES Transformer: Pre-trained Molecular Fingerprint for Low Data Drug Discovery](https://mlsanity.com/paper/1911.04738/) (2019)
- [Molecular Property Prediction: A Multilevel Quantum Interactions Modeling Perspective](https://mlsanity.com/paper/1906.11081/) (2019)
- [Massively Multitask Networks for Drug Discovery](https://mlsanity.com/paper/1502.02072/) (2015)
- [Exploiting Neuron and Synapse Filter Dynamics in Spatial Temporal Learning of Deep Spiking Neural Network](https://mlsanity.com/paper/2003.02944/) (2020)
- [Path-Augmented Graph Transformer Network](https://mlsanity.com/paper/1905.12712/) (2019)
- [GResNet: Graph Residual Network for Reviving Deep GNNs from Suspended Animation](https://mlsanity.com/paper/1909.05729/) (2019)
- [Edge Attention-based Multi-Relational Graph Convolutional Networks](https://mlsanity.com/paper/1802.04944/) (2018)

## Built on by (most cited first, 20 of 20)

- [GeomGCL: Geometric Graph Contrastive Learning for Molecular Property Prediction](https://mlsanity.com/paper/2109.11730/) (2021)
- [Molecular Contrastive Learning with Chemical Element Knowledge Graph](https://mlsanity.com/paper/2112.00544/) (2021)
- [Molecular Property Prediction Based on Graph Structure Learning](https://mlsanity.com/paper/2312.16855/) (2023)
- [Gradformer: Graph Transformer with Exponential Decay](https://mlsanity.com/paper/2404.15729/) (2024)
- [Motif-based Graph Representation Learning with Application to Chemical Molecules](https://mlsanity.com/paper/2208.04529/) (2022)
- [3D Graph Contrastive Learning for Molecular Property Prediction](https://mlsanity.com/paper/2208.06360/) (2022)
- [Retroformer: Pushing the Limits of Interpretable End-to-end Retrosynthesis Transformer](https://mlsanity.com/paper/2201.12475/) (2022)
- [Molecular Joint Representation Learning via Multi-modal Information](https://mlsanity.com/paper/2211.14042/) (2022)
- [Collaborative Graph Neural Networks for Attributed Network Embedding](https://mlsanity.com/paper/2307.11981/) (2023)
- [Efficient Sharpness-Aware Minimization for Molecular Graph Transformer Models](https://mlsanity.com/paper/2406.13137/) (2024)
- [GEM-2: Next Generation Molecular Property Prediction Network by Modeling Full-range Many-body Interactions](https://mlsanity.com/paper/2208.05863/) (2022)
- [Advancing Drug Discovery with Enhanced Chemical Understanding via Asymmetric Contrastive Multimodal Learning](https://mlsanity.com/paper/2311.06456/) (2023)
- [Knowledge-informed Molecular Learning: A Survey on Paradigm Transfer](https://mlsanity.com/paper/2202.10587/) (2022)
- [Substructure-Atom Cross Attention for Molecular Representation Learning](https://mlsanity.com/paper/2210.08243/) (2022)
- [Knowledge-aware contrastive heterogeneous molecular graph learning](https://mlsanity.com/paper/2502.11711/) (2025)
- [Geometric Multi-color Message Passing Graph Neural Networks for Blood-brain Barrier Permeability Prediction](https://mlsanity.com/paper/2507.18926/) (2025)
- [T- Hop: A framework for studying the importance path information in molecular graphs for chemical property prediction](https://mlsanity.com/paper/2407.14270/) (2024)
- [Graph Fourier Transformer with Structure-Frequency Information](https://mlsanity.com/paper/2504.19740/) (2025)
- [Optimal message passing for molecular prediction is simple, attentive and spatial](https://mlsanity.com/paper/2509.10871/) (2025)
- [MMGNN: Multi-level, multi-color graph neural networks for molecular property prediction](https://mlsanity.com/paper/2606.20906/) (2026)

## Abstract (arXiv)

Constructing appropriate representations of molecules lies at the core of numerous tasks such as material science, chemistry and drug designs. Recent researches abstract molecules as attributed graphs and employ graph neural networks (GNN) for molecular representation learning, which have made remarkable achievements in molecular graph modeling. Albeit powerful, current models either are based on local aggregation operations and thus miss higher-order graph properties or focus on only node information without fully using the edge information. For this sake, we propose a Communicative Message Passing Transformer (CoMPT) neural network to improve the molecular graph representation by reinforcing message interactions between nodes and edges based on the Transformer architecture. Unlike the previous transformer-style GNNs that treat molecules as fully connected graphs, we introduce a message diffusion mechanism to leverage the graph connectivity inductive bias and reduce the message enrichment explosion. Extensive experiments demonstrated that the proposed model obtained superior performances (around 4% on average) against state-of-the-art baselines on seven chemical property datasets (graph-level tasks) and two chemical shift datasets (node-level tasks). Further visualization studies also indicated a better representation capacity achieved by our model.