Pretraining Graph Transformers with Atom-in-a-Molecule Quantum Properties for Improved ADMET Modeling
Not on the map. Why?
Abstract (arXiv)
We evaluate the impact of pretraining Graph Transformer architectures on atom-level quantum-mechanical features for the modeling of absorption, distribution, metabolism, excretion, and toxicity (ADMET) properties of drug-like compounds. We compare this pretraining strategy with two others: one based on molecular quantum properties (specifically the HOMO-LUMO gap) and one using a self-supervised atom masking technique. After fine-tuning on Therapeutic Data Commons ADMET datasets, we evaluate the performance improvement in the different models observing that models pretrained with atomic quantum mechanical properties produce in general better results. We then analyse the latent representations and observe that the supervised strategies preserve the pretraining information after finetuning and that different pretrainings produce different trends in latent expressivity across layers. Furthermore, we find that models pretrained on atomic quantum mechanical properties capture more low-frequency laplacian eigenmodes of the input graph via the attention weights and produce better representations of atomic environments within the molecule. Application of the analysis to a much larger non-public dataset for microsomal clearance illustrates generalizability of the studied indicators. In this case the performances of the models are in accordance with the representation analysis and highlight, especially for the case of masking pretraining and atom-level quantum property pretraining, how model types with similar performance on public benchmarks can have different performances on large scale pharmaceutical data.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 31 of its 81 references that are in the map. First posted on arXiv 10 Oct 2024.
Builds on
31 of 81 references.
References · 31
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
- 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
- 2016Convolutional Neural Networks on Graphs with Fast Localized Spectral Filteringcited by 2,089
- 2013Spectral Networks and Locally Connected Networks on Graphscited by 1,102
- 2019Strategies for Pre-training Graph Neural Networkscited by 971
- 2020Quantifying Attention Flow in Transformerscited by 775
- 2021E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentialscited by 708
- 2017SchNet: A continuous-filter convolutional neural network for modeling quantum interactionscited by 571
- 2021Understanding over-squashing and bottlenecks on graphs via curvaturecited by 478
- 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
19 more
- 2020The Open Catalyst 2020 (OC20) Dataset and Community Challengescited by 246
- 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 141
- 2022A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Languagecited by 131
- 2020Molecular representation learning with language models and domain-relevant auxiliary taskscited by 94
- 2023Attending to Graph Transformerscited by 83
- 2018Less is more: sampling chemical space with active learningcited by 82
- 2021Graph Representation Learning in Biomedicinecited by 80
- 2022Revisiting Over-smoothing in BERT from the Perspective of Graphcited by 72
- 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
- 2023Towards Foundational Models for Molecular Learning on Large-Scale Multi-Task Datasetscited by 39
- 2023From Molecules to Materials: Pre-training Large Generalizable Models for Atomic Property Predictioncited by 39
- 2022A Systematic Survey of Chemical Pre-trained Modelscited by 34
- 2024textttMiniMol: A Parameter-Efficient Foundation Model for Molecular Learningcited by 20
- 2023Rank Collapse Causes Over-Smoothing and Over-Correlation in Graph Neural Networkscited by 18
- 2023Denoise Pretraining on Nonequilibrium Molecules for Accurate and Transferable Neural Potentialscited by 11
- 2024Multi-Modal Representation Learning for Molecular Property Prediction: Sequence, Graph, Geometrycited by 10
- 2024Preventing Representational Rank Collapse in MPNNs by Splitting the Computational Graphcited by 6
- 2021Benchmarking Accuracy and Generalizability of Four Graph Neural Networks Using Large In Vitro ADME Datasets from Different Chemical Spacescited by 5
- 2024Analysis of Atom-level pretraining with Quantum Mechanics (QM) data for Graph Neural Networks Molecular property modelscited by 1
Built on by
Citing papers per year: 2025: 2; 2026: 1.
3 papers.