Pre-training with Graph Transformers
Not on the map. Why?
Abstract (arXiv)
This article investigates pre-training strategies for graph transformers in the biochemistry domain. By conducting comprehensive experiments, the study reveals that supervised pre-training using computed properties as labels provides the highest performance gain on downstream tasks. The results also highlight the importance of constraining model capacity to mitigate overfitting in graph transformers.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 13 of its 22 references that are in the map. First posted on arXiv 12 Sep 2026.
Builds on
13 of 22 references.
References · 13
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2018Deep Graph Infomaxcited by 1,101
- 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,037
- 2019Strategies for Pre-training Graph Neural Networkscited by 964
- 2022Recipe for a General, Powerful, Scalable Graph Transformercited by 702
- 2019Graph Transformer Networkscited by 520
- 2020Inductive Representation Learning on Temporal Graphscited by 454
- 2021Graph Neural Networks with Learnable Structural and Positional Representationscited by 326
- 2018Dynamic Graph Representation Learning via Self-Attention Networkscited by 39
1 more
- 2023Bi-Level Attention Graph Neural Networkscited by 6
Built on by
Not cited yet.