mlsanityData to 7 Oct 2026

Pre-training Transformers for Molecular Property Prediction Using Reaction Prediction

arXiv 2207.02724 ↗First posted 6 Jul 2022cs.LG physics.chem-ph q-bio.BMcited by 5

Not on the map. Why?

Abstract (arXiv)

Molecular property prediction is essential in chemistry, especially for drug discovery applications. However, available molecular property data is often limited, encouraging the transfer of information from related data. Transfer learning has had a tremendous impact in fields like Computer Vision and Natural Language Processing signaling for its potential in molecular property prediction. We present a pre-training procedure for molecular representation learning using reaction data and use it to pre-train a SMILES Transformer. We fine-tune and evaluate the pre-trained model on 12 molecular property prediction tasks from MoleculeNet within physical chemistry, biophysics, and physiology and show a statistically significant positive effect on 5 of the 12 tasks compared to a non-pre-trained baseline model.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 23 of its 41 references that are in the map. First posted on arXiv 6 Jul 2022.

Builds on

23 of 41 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  5. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  6. 2019A Comprehensive Survey on Transfer Learningcited by 1,154
  7. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,037
  8. 2019Strategies for Pre-training Graph Neural Networkscited by 964
  9. 2020ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Predictioncited by 332
  10. 2021Do Transformers Really Perform Bad for Graph Representation?cited by 318
  11. 2021Pre-training Molecular Graph Representation with 3D Geometrycited by 248
  12. 2015Order Matters: Sequence to sequence for setscited by 236
11 more
  1. 2018Molecular Transformer - A Model for Uncertainty-Calibrated Chemical Reaction Predictioncited by 225
  2. 2021ChemRL-GEM: Geometry Enhanced Molecular Representation Learning for Property Predictioncited by 204
  3. 2021Motif-based Graph Self-Supervised Learning for Molecular Property Predictioncited by 159
  4. 20213D Infomax improves GNNs for Molecular Property Predictioncited by 146
  5. 2020Molecule Attention Transformercited by 92
  6. 2020Molecular representation learning with language models and domain-relevant auxiliary taskscited by 91
  7. 2017SMILES Enumeration as Data Augmentation for Neural Network Modeling of Moleculescited by 58
  8. 2017"Found in Translation": Predicting Outcomes of Complex Organic Chemistry Reactions using Neural Sequence-to-Sequence Modelscited by 54
  9. 2018Universal Language Model Fine-tuning for Text Classificationcited by 50
  10. 2021Chemical-Reaction-Aware Molecule Representation Learningcited by 36
  11. 2020Graph-Aware Transformer: Is Attention All Graphs Need?cited by 6

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 3.

5 papers.

  1. 2024Impact of Domain Knowledge and Multi-Modality on Intelligent Molecular Property Prediction: A Systematic Surveycited by 6
  2. 2023Enhancing Molecular Property Prediction via Mixture of Collaborative Expertscited by 2
  3. 2024Pre-trained Molecular Language Models with Random Functional Group Maskingcited by 2
  4. 2023OL-Transformer: A Fast and Universal Surrogate Simulator for Optical Multilayer Thin Film Structurescited by 1
  5. 2024Contextual Molecule Representation Learning from Chemical Reaction Knowledgecited by 1