mlsanityData to 7 Oct 2026

Molecule Attention Transformer

arXiv 2002.08264 ↗First posted 19 Feb 2020cs.LG physics.comp-ph stat.MLcited by 92

Not on the map. Why?

Abstract (arXiv)

Designing a single neural network architecture that performs competitively across a range of molecule property prediction tasks remains largely an open challenge, and its solution may unlock a widespread use of deep learning in the drug discovery industry. To move towards this goal, we propose Molecule Attention Transformer (MAT). Our key innovation is to augment the attention mechanism in Transformer using inter-atomic distances and the molecular graph structure. Experiments show that MAT performs competitively on a diverse set of molecular prediction tasks. Most importantly, with a simple self-supervised pretraining, MAT requires tuning of only a few hyperparameter values to achieve state-of-the-art performance on downstream tasks. Finally, we show that attention weights learned by MAT are interpretable from the chemical point of view.

Key facts. Cited by 92 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 15; 2022: 16; 2023: 23; 2024: 17; 2025: 16; 2026: 5.

92 papers.

Sort:

By area:

  1. 2021Do Transformers Really Perform Bad for Graph Representation?cited by 318
  2. 2021ChemRL-GEM: Geometry Enhanced Molecular Representation Learning for Property Predictioncited by 204
  3. 2022Graph neural networks for materials science and chemistrycited by 181
  4. 2022XAI for Transformers: Better Explanations through Conservative Propagationcited by 100
  5. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  6. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 96
  7. 2024Scientific Large Language Models: A Survey on Biological & Chemical Domainscited by 80
  8. 2022One Transformer Can Understand Both 2D & 3D Molecular Datacited by 77
  9. 2023GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learningcited by 70
  10. 2021Permutation invariant graph-to-sequence model for template-free retrosynthesis and reaction predictioncited by 66
  11. 2021GeomGCL: Geometric Graph Contrastive Learning for Molecular Property Predictioncited by 61
  12. 2021Structure-aware Interactive Graph Neural Networks for the Prediction of Protein-Ligand Binding Affinitycited by 60
38 more
  1. 2021Molformer: Motif-based Transformer on 3D Heterogeneous Molecular Graphscited by 44
  2. 2023Enhancing Activity Prediction Models in Drug Discovery with the Ability to Understand Human Languagecited by 44
  3. 2022KPGT: Knowledge-Guided Pre-training of Graph Transformer for Molecular Property Predictioncited by 31
  4. 2021Dual-view Molecule Pre-trainingcited by 27
  5. 2023G-Adapter: Towards Structure-Aware Parameter-Efficient Transfer Learning for Graph Transformer Networkscited by 25
  6. 2024Large Language Models for Manufacturingcited by 24
  7. 2023Context-enriched molecule representations improve few-shot drug discoverycited by 23
  8. 2022Meta-learning Adaptive Deep Kernel Gaussian Processes for Molecular Property Predictioncited by 21
  9. 2021Learning Attributed Graph Representations with Communicative Message Passing Transformercited by 20
  10. 2025Benchmarking Pretrained Molecular Embedding Models For Molecular Representation Learningcited by 15
  11. 2023Are More Layers Beneficial to Graph Transformers?cited by 14
  12. 2023Molecular Property Prediction Based on Graph Structure Learningcited by 13
  13. 2024ReactZyme: A Benchmark for Enzyme-Reaction Predictioncited by 13
  14. 2025A Survey of Graph Transformers: Architectures, Theories and Applicationscited by 12
  15. 2023Activity Cliff Prediction: Dataset and Benchmarkcited by 11
  16. 2021Attentive Walk-Aggregating Graph Neural Networkscited by 10
  17. 2022Graph Masked Autoencoders with Transformerscited by 10
  18. 2023Deep Learning Methods for Small Molecule Drug Discovery: A Surveycited by 10
  19. 2023A Universal Framework for Accurate and Efficient Geometric Deep Learning of Molecular Systemscited by 10
  20. 2021Efficient Softmax Approximation for Deep Neural Networks with Attention Mechanismcited by 9
  21. 2021Geometric Transformer for End-to-End Molecule Properties Predictioncited by 8
  22. 2022ProGReST: Prototypical Graph Regression Soft Trees for Molecular Property Predictioncited by 8
  23. 2023Difficulty in chirality recognition for Transformer architectures learning chemical structures from stringcited by 7
  24. 2023Multi-task Bioassay Pre-training for Protein-ligand Binding Affinity Predictioncited by 7
  25. 2023In-Context Learning for Few-Shot Molecular Property Predictioncited by 7
  26. 2023Explainability Techniques for Chemical Language Modelscited by 6
  27. 2023ChiENN: Embracing Molecular Chirality with Graph Neural Networkscited by 6
  28. 20233D-Mol: A Novel Contrastive Learning Framework for Molecular Property Prediction with 3D Informationcited by 6
  29. 2024Impact of Domain Knowledge and Multi-Modality on Intelligent Molecular Property Prediction: A Systematic Surveycited by 6
  30. 2022Pre-training Transformers for Molecular Property Prediction Using Reaction Predictioncited by 5
  31. 2022GEM-2: Next Generation Molecular Property Prediction Network by Modeling Full-range Many-body Interactionscited by 5
  32. 2022Expressing Multivariate Time Series as Graphs with Time Series Attention Transformercited by 5
  33. 2024EnzymeFlow: Generating Reaction-specific Enzyme Catalytic Pockets through Flow Matching and Co-Evolutionary Dynamicscited by 5
  34. 2026Molecular Representations in Implicit Functional Space via Hyper-Networkscited by 5
  35. 2023Automated patent extraction powers generative modeling in focused chemical spacescited by 4
  36. 2023Uni-QSAR: an Auto-ML Tool for Molecular Property Predictioncited by 4
  37. 2023PACIA: Parameter-Efficient Adapter for Few-Shot Molecular Property Predictioncited by 4
  38. 2023Atom-Motif Contrastive Transformer for Molecular Property Predictioncited by 4

50 most cited of 92 All 92 (JSON)