# Molecule Attention Transformer

arXiv: https://arxiv.org/abs/2002.08264 · HTML: https://mlsanity.com/paper/2002.08264/

## Key facts

- Cited by 92 papers in the map (data through 13 Oct 2026).
- First posted on arXiv 19 Feb 2020.

## Built on by (most cited first, 30 of 92)

- [Do Transformers Really Perform Bad for Graph Representation?](https://mlsanity.com/paper/2106.05234/) (2021)
- [ChemRL-GEM: Geometry Enhanced Molecular Representation Learning for Property Prediction](https://mlsanity.com/paper/2106.06130/) (2021)
- [Graph neural networks for materials science and chemistry](https://mlsanity.com/paper/2208.09481/) (2022)
- [XAI for Transformers: Better Explanations through Conservative Propagation](https://mlsanity.com/paper/2202.07304/) (2022)
- [A Review of Large Language Models and Autonomous Agents in Chemistry](https://mlsanity.com/paper/2407.01603/) (2024)
- [A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery](https://mlsanity.com/paper/2406.10833/) (2024)
- [Scientific Large Language Models: A Survey on Biological & Chemical Domains](https://mlsanity.com/paper/2401.14656/) (2024)
- [One Transformer Can Understand Both 2D & 3D Molecular Data](https://mlsanity.com/paper/2210.01765/) (2022)
- [GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learning](https://mlsanity.com/paper/2306.13089/) (2023)
- [Permutation invariant graph-to-sequence model for template-free retrosynthesis and reaction prediction](https://mlsanity.com/paper/2110.09681/) (2021)
- [GeomGCL: Geometric Graph Contrastive Learning for Molecular Property Prediction](https://mlsanity.com/paper/2109.11730/) (2021)
- [Structure-aware Interactive Graph Neural Networks for the Prediction of Protein-Ligand Binding Affinity](https://mlsanity.com/paper/2107.10670/) (2021)
- [Molformer: Motif-based Transformer on 3D Heterogeneous Molecular Graphs](https://mlsanity.com/paper/2110.01191/) (2021)
- [Enhancing Activity Prediction Models in Drug Discovery with the Ability to Understand Human Language](https://mlsanity.com/paper/2303.03363/) (2023)
- [KPGT: Knowledge-Guided Pre-training of Graph Transformer for Molecular Property Prediction](https://mlsanity.com/paper/2206.03364/) (2022)
- [Dual-view Molecule Pre-training](https://mlsanity.com/paper/2106.10234/) (2021)
- [G-Adapter: Towards Structure-Aware Parameter-Efficient Transfer Learning for Graph Transformer Networks](https://mlsanity.com/paper/2305.10329/) (2023)
- [Context-enriched molecule representations improve few-shot drug discovery](https://mlsanity.com/paper/2305.09481/) (2023)
- [Large Language Models for Manufacturing](https://mlsanity.com/paper/2410.21418/) (2024)
- [Meta-learning Adaptive Deep Kernel Gaussian Processes for Molecular Property Prediction](https://mlsanity.com/paper/2205.02708/) (2022)
- [Learning Attributed Graph Representations with Communicative Message Passing Transformer](https://mlsanity.com/paper/2107.08773/) (2021)
- [Benchmarking Pretrained Molecular Embedding Models For Molecular Representation Learning](https://mlsanity.com/paper/2508.06199/) (2025)
- [Are More Layers Beneficial to Graph Transformers?](https://mlsanity.com/paper/2303.00579/) (2023)
- [Molecular Property Prediction Based on Graph Structure Learning](https://mlsanity.com/paper/2312.16855/) (2023)
- [ReactZyme: A Benchmark for Enzyme-Reaction Prediction](https://mlsanity.com/paper/2408.13659/) (2024)
- [A Survey of Graph Transformers: Architectures, Theories and Applications](https://mlsanity.com/paper/2502.16533/) (2025)
- [Activity Cliff Prediction: Dataset and Benchmark](https://mlsanity.com/paper/2302.07541/) (2023)
- [Attentive Walk-Aggregating Graph Neural Networks](https://mlsanity.com/paper/2110.02667/) (2021)
- [Graph Masked Autoencoders with Transformers](https://mlsanity.com/paper/2202.08391/) (2022)
- [Deep Learning Methods for Small Molecule Drug Discovery: A Survey](https://mlsanity.com/paper/2303.00313/) (2023)

## Abstract (arXiv)

Designing a single neural network architecture that performs competitively across a range of molecule property prediction tasks remains largely an open challenge, and its solution may unlock a widespread use of deep learning in the drug discovery industry. To move towards this goal, we propose Molecule Attention Transformer (MAT). Our key innovation is to augment the attention mechanism in Transformer using inter-atomic distances and the molecular graph structure. Experiments show that MAT performs competitively on a diverse set of molecular prediction tasks. Most importantly, with a simple self-supervised pretraining, MAT requires tuning of only a few hyperparameter values to achieve state-of-the-art performance on downstream tasks. Finally, we show that attention weights learned by MAT are interpretable from the chemical point of view.