mlsanityData to 7 Oct 2026

A Systematic Survey of Chemical Pre-trained Models

arXiv 2210.16484 ↗First posted 29 Oct 2022cs.LG cs.AIcited by 34

Not on the map. Why?

Abstract (arXiv)

Deep learning has achieved remarkable success in learning representations for molecules, which is crucial for various biochemical applications, ranging from property prediction to drug design. However, training Deep Neural Networks (DNNs) from scratch often requires abundant labeled molecules, which are expensive to acquire in the real world. To alleviate this issue, tremendous efforts have been devoted to Molecular Pre-trained Models (CPMs), where DNNs are pre-trained using large-scale unlabeled molecular databases and then fine-tuned over specific downstream tasks. Despite the prosperity, there lacks a systematic review of this fast-growing field. In this paper, we present the first survey that summarizes the current progress of CPMs. We first highlight the limitations of training molecular representation models from scratch to motivate CPM studies. Next, we systematically review recent advances on this topic from several key perspectives, including molecular descriptors, encoder architectures, pre-training strategies, and applications. We also highlight the challenges and promising avenues for future research, providing a useful resource for both machine learning and scientific communities.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 55 of its 90 references that are in the map. First posted on arXiv 29 Oct 2022.

Builds on

55 of 90 references.

References · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  7. 2017Graph Attention Networkscited by 8,854
  8. 2018How Powerful are Graph Neural Networks?cited by 4,355
  9. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  10. 2020Graph Contrastive Learning with Augmentationscited by 1,203
  11. 2018Deep Graph Infomaxcited by 1,101
  12. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,037
38 more
  1. 2019Strategies for Pre-training Graph Neural Networkscited by 964
  2. 2021E(n) Equivariant Graph Neural Networkscited by 951
  3. 2020Learning to Simulate Complex Physics with Graph Networkscited by 867
  4. 2020Contrastive Multi-View Representation Learning on Graphscited by 651
  5. 2017SchNet: A continuous-filter convolutional neural network for modeling quantum interactionscited by 569
  6. 2020Principal Neighbourhood Aggregation for Graph Netscited by 477
  7. 2022GraphMAE: Self-Supervised Masked Graph Autoencoderscited by 443
  8. 2019InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximizationcited by 441
  9. 2021Equivariant message passing for the prediction of tensorial properties and molecular spectracited by 438
  10. 2021OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphscited by 328
  11. 2020GPT-GNN: Generative Pre-Training of Graph Neural Networkscited by 302
  12. 2021Graph Contrastive Learning Automatedcited by 287
  13. 2021Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Developmentcited by 258
  14. 2021Pre-training Molecular Graph Representation with 3D Geometrycited by 248
  15. 2016Molecular Graph Convolutions: Moving Beyond Fingerprintscited by 238
  16. 2020GEOM: Energy-annotated molecular conformations for property prediction and molecular generationcited by 238
  17. 2021ChemRL-GEM: Geometry Enhanced Molecular Representation Learning for Property Predictioncited by 204
  18. 2021Large-Scale Chemical Language Representations Capture Molecular Structure and Propertiescited by 203
  19. 2021Adversarial Graph Augmentation to Improve Graph Contrastive Learningcited by 200
  20. 2022Translation between Molecules and Natural Languagecited by 173
  21. 2022SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentationcited by 171
  22. 2021Motif-based Graph Self-Supervised Learning for Molecular Property Predictioncited by 159
  23. 20213D Infomax improves GNNs for Molecular Property Predictioncited by 146
  24. 2021Self-supervised Graph-level Representation Learning with Local and Global Structurecited by 125
  25. 2022Pre-training via Denoising for Molecular Property Predictioncited by 124
  26. 2021InfoGCL: Information-Aware Graph Contrastive Learningcited by 105
  27. 2019SMILES Transformer: Pre-trained Molecular Fingerprint for Low Data Drug Discoverycited by 89
  28. 2021SE(3) Equivariant Graph Neural Networks with Complete Local Framescited by 75
  29. 2021ProGCL: Rethinking Hard Negative Mining in Graph Contrastive Learningcited by 72
  30. 2022Molecular Geometry Pretraining with SE(3)-Invariant Denoising Distance Matchingcited by 67
  31. 2022MolGenSurvey: A Systematic Survey in Machine Learning Models for Molecule Designcited by 66
  32. 2021GeomGCL: Geometric Graph Contrastive Learning for Molecular Property Predictioncited by 61
  33. 2022Does GNN Pretraining Help Molecular Representation?cited by 59
  34. 2021Molecular Contrastive Learning with Chemical Element Knowledge Graphcited by 50
  35. 2022Unified 2D and 3D Pre-Training of Molecular Representationscited by 45
  36. 2022Improving Molecular Contrastive Learning via Faulty Negative Mitigation and Decomposed Fragment Contrastcited by 37
  37. 2022Energy-Motivated Equivariant Pretraining for 3D Molecular Graphscited by 37
  38. 2021Molecule3D: A Benchmark for Predicting 3D Geometries from Molecular Graphscited by 31

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 14; 2025: 3; 2026: 4.

34 papers.

Sort:

By area:

  1. 2023Graph Foundation Models: Concepts, Opportunities and Challengescited by 101
  2. 2023Towards Foundational Models for Molecular Learning on Large-Scale Multi-Task Datasetscited by 37
  3. 2024A Survey of Geometric Graph Neural Networks: Data Structures, Models and Applicationscited by 28
  4. 2023Uncovering Neural Scaling Laws in Molecular Representation Learningcited by 23
  5. 2023Fragment-based Pretraining and Finetuning on Molecular Graphscited by 20
  6. 2023Graph Meets LLMs: Towards Large Graph Modelscited by 19
  7. 2024Large Language Models are In-Context Molecule Learnerscited by 18
  8. 2024ChemDFM-X: Towards Large Multimodal Model for Chemistrycited by 16
  9. 2024From Words to Molecules: A Survey of Large Language Models in Chemistrycited by 14
  10. 2023Search to Fine-tune Pre-trained Graph Neural Networks for Graph-level Taskscited by 13
  11. 2023ReactionT5: a large-scale pre-trained model towards application of limited reaction datacited by 13
  12. 2024Geometry Informed Tokenization of Molecules for Language Model Generationcited by 10
22 more
  1. 2023Learning Over Molecular Conformer Ensembles: Datasets and Benchmarkscited by 9
  2. 2024PeaTMOSS: A Dataset and Initial Analysis of Pre-Trained Models in Open-Source Softwarecited by 9
  3. 2023M2Hub: Unlocking the Potential of Machine Learning for Materials Discoverycited by 8
  4. 2024Representing Molecules as Random Walks Over Interpretable Grammarscited by 7
  5. 2023UniMAP: Universal SMILES-Graph Representation Learningcited by 6
  6. 2023Improving Self-supervised Molecular Representation Learning using Persistent Homologycited by 6
  7. 20253DMolFormer: A Dual-channel Framework for Structure-based Drug Discoverycited by 4
  8. 2026Agentic reinforcement learning empowers next-generation chemical language models for molecular design and synthesiscited by 4
  9. 2023Why Deep Models Often cannot Beat Non-deep Counterparts on Molecular Property Prediction?cited by 3
  10. 2024MolMix: A Simple Yet Effective Baseline for Multimodal Molecular Representation Learningcited by 3
  11. 2024Pretraining Graph Transformers with Atom-in-a-Molecule Quantum Properties for Improved ADMET Modelingcited by 3
  12. 2024FlexMol: A Flexible Toolkit for Benchmarking Molecular Relational Learningcited by 3
  13. 2023De Novo Drug Design with Joint Transformerscited by 2
  14. 2024Analysis of Atom-level pretraining with Quantum Mechanics (QM) data for Graph Neural Networks Molecular property modelscited by 1
  15. 2024A Strong Baseline for Molecular Few-Shot Learningcited by 0
  16. 2024Relating-Up: Advancing Graph Neural Networks through Inter-Graph Relationshipscited by 0
  17. 2024MolFusion: Multimodal Fusion Learning for Molecular Representations via Multi-granularity Viewscited by 0
  18. 2025Modeling All-Atom Glycan Structures via Hierarchical Message Passing and Multi-Scale Pre-trainingcited by 0
  19. 2025A Conformation-Centric Generative Foundation Model for Linear Polymer Modeling and Designcited by 0
  20. 2026Provable Target Sample Complexity Improvements as Pre-Trained Models Scalecited by 0
  21. 2026AtomComposer: Discovering Chemical Space from First Principles with Reinforcement Learningcited by 0
  22. 2026TMCS: Tool-Grounded Multi-Agent Reasoning for Compositional Chemical Problem Solvingcited by 0