mlsanityData to 7 Oct 2026

When Does Global Attention Help? A Unified Empirical Study on Atomistic Graph Learning

arXiv 2510.05583 ↗First posted 7 Oct 2025cs.LG cs.DCcited by 0

Not on the map. Why?

Abstract (arXiv)

Graph neural networks (GNNs) are widely used as surrogates for costly experiments and first-principles simulations to study the behavior of compounds at atomistic scale, and their architectural complexity is constantly increasing to enable the modeling of complex physics. While most recent GNNs combine more traditional message passing neural networks (MPNNs) layers to model short-range interactions with more advanced graph transformers (GTs) with global attention mechanisms to model long-range interactions, it is still unclear when global attention mechanisms provide real benefits over well-tuned MPNN layers due to inconsistent implementations, features, or hyperparameter tuning. We introduce the first unified, reproducible benchmarking framework - built on HydraGNN - that enables seamless switching among four controlled model classes: MPNN, MPNN with chemistry/topology encoders, GPS-style hybrids of MPNN with global attention, and fully fused local-global models with encoders. Using seven diverse open-source datasets for benchmarking across regression and classification tasks, we systematically isolate the contributions of message passing, global attention, and encoder-based feature augmentation. Our study shows that encoder-augmented MPNNs form a robust baseline, while fused local-global models yield the clearest benefits for properties governed by long-range interaction effects. We further quantify the accuracy-compute trade-offs of attention, reporting its overhead in memory. Together, these results establish the first controlled evaluation of global attention in atomistic graph learning and provide a reproducible testbed for future model development.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 42 of its 91 references that are in the map. First posted on arXiv 7 Oct 2025.

Builds on

42 of 91 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  3. 2017Graph Attention Networkscited by 8,854
  4. 2017Inductive Representation Learning on Large Graphscited by 6,284
  5. 2018How Powerful are Graph Neural Networks?cited by 4,355
  6. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
  7. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,906
  8. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
  9. 2018Graph Neural Networks: A Review of Methods and Applicationscited by 1,571
  10. 2017Deep Setscited by 1,394
  11. 2018Relational inductive biases, deep learning, and graph networkscited by 1,341
  12. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
30 more
  1. 2018Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learningcited by 1,099
  2. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,037
  3. 2019Strategies for Pre-training Graph Neural Networkscited by 964
  4. 2021E(n) Equivariant Graph Neural Networkscited by 951
  5. 2021How Attentive are Graph Attention Networks?cited by 818
  6. 2015Convolutional Networks on Graphs for Learning Molecular Fingerprintscited by 751
  7. 2018Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point cloudscited by 735
  8. 2022Recipe for a General, Powerful, Scalable Graph Transformercited by 702
  9. 2021E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentialscited by 701
  10. 2020A Generalization of Transformer Networks to Graphscited by 653
  11. 2020On the Bottleneck of Graph Neural Networks and its Practical Implicationscited by 624
  12. 2020SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networkscited by 602
  13. 2017SchNet: A continuous-filter convolutional neural network for modeling quantum interactionscited by 569
  14. 2020Directional Message Passing for Molecular Graphscited by 556
  15. 2021Understanding over-squashing and bottlenecks on graphs via curvaturecited by 471
  16. 2021Rethinking Graph Transformers with Spectral Attentioncited by 468
  17. 2021Equivariant message passing for the prediction of tensorial properties and molecular spectracited by 438
  18. 2021OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphscited by 328
  19. 2016Molecular Graph Convolutions: Moving Beyond Fingerprintscited by 238
  20. 2023EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representationscited by 201
  21. 2021Global Self-Attention as a Replacement for Graph Convolutioncited by 113
  22. 2020Molecule Attention Transformercited by 92
  23. 2023Where Did the Gap Go? Reassessing the Long-Range Graph Benchmarkcited by 83
  24. 2015Massively Multitask Networks for Drug Discoverycited by 49
  25. 2023Boosting Multitask Learning on Graphs through Higher-Order Task Affinitiescited by 15
  26. 2024Scalable Training of Trustworthy and Energy-Efficient Predictive Graph Foundation Models for Atomistic Materials Modeling: A Case Study with HydraGNNcited by 8
  27. 2019Multitask Learning On Graph Neural Networks Applied To Molecular Property Predictionscited by 6
  28. 2022Multi-task graph neural networks for simultaneous prediction of global and atomic properties in ferromagnetic systemscited by 6
  29. 2022Asynchronous Decentralized Bayesian Optimization for Large Scale Hyperparameter Optimizationcited by 6
  30. 2022Scalable training of graph convolutional neural networks for fast and accurate predictions of HOMO-LUMO gap in moleculescited by 3

Built on by

Not cited yet.