mlsanityData to 14 Oct 2026

Pre-training via Denoising for Molecular Property Prediction

arXiv 2206.00133 ↗First posted 31 May 2022cs.LG q-bio.BM stat.MLcited by 124

Not on the map. Why?

Abstract (arXiv)

Many important problems involving molecular property prediction from 3D structures have limited data, posing a generalization challenge for neural networks. In this paper, we describe a pre-training technique based on denoising that achieves a new state-of-the-art in molecular property prediction by utilizing large datasets of 3D molecular structures at equilibrium to learn meaningful representations for downstream tasks. Relying on the well-known link between denoising autoencoders and score-matching, we show that the denoising objective corresponds to learning a molecular force field -- arising from approximating the Boltzmann distribution with a mixture of Gaussians -- directly from equilibrium structures. Our experiments demonstrate that using this pre-training objective significantly improves performance on multiple benchmarks, achieving a new state-of-the-art on the majority of targets in the widely used QM9 dataset. Our analysis then provides practical insights into the effects of different factors -- dataset sizes, model size and architecture, and the choice of upstream and downstream datasets -- on pre-training.

Key facts. Cited by 124 papers in the map (data through 14 Oct 2026). Builds on 62 of its 89 references that are in the map. First posted on arXiv 31 May 2022.

Builds on

62 of 89 references.

References · 62

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  6. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  7. 2016Variational Graph Auto-Encoderscited by 1,382
  8. 2018Relational inductive biases, deep learning, and graph networkscited by 1,360
  9. 2020Graph Contrastive Learning with Augmentationscited by 1,208
  10. 2018Deep Graph Infomaxcited by 1,106
  11. 2016Geometric deep learning: going beyond Euclidean datacited by 1,048
  12. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,043
38 more
  1. 2019Strategies for Pre-training Graph Neural Networkscited by 971
  2. 2021E(n) Equivariant Graph Neural Networkscited by 965
  3. 2020Learning to Simulate Complex Physics with Graph Networkscited by 880
  4. 2020Learning Mesh-Based Simulation with Graph Networkscited by 868
  5. 2018Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point cloudscited by 746
  6. 2021E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentialscited by 708
  7. 2022Equivariant Diffusion for Molecule Generation in 3Dcited by 677
  8. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 618
  9. 2020SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networkscited by 607
  10. 2017SchNet: A continuous-filter convolutional neural network for modeling quantum interactionscited by 571
  11. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  12. 2020Directional Message Passing for Molecular Graphscited by 561
  13. 2019Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological Viewcited by 545
  14. 2022GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generationcited by 516
  15. 2016Interaction Networks for Learning about Objects, Relations and Physicscited by 509
  16. 2019InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximizationcited by 441
  17. 2021Equivariant message passing for the prediction of tensorial properties and molecular spectracited by 441
  18. 20183D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Datacited by 354
  19. 2021GemNet: Universal Directional Graph Neural Networks for Moleculescited by 343
  20. 2021Do Transformers Really Perform Bad for Graph Representation?cited by 320
  21. 2020GPT-GNN: Generative Pre-Training of Graph Neural Networkscited by 303
  22. 2019PairNorm: Tackling Oversmoothing in GNNscited by 297
  23. 2021Graph Self-Supervised Learning: A Surveycited by 289
  24. 2019Cormorant: Covariant Molecular Neural Networkscited by 263
  25. 2020Fast and Uncertainty-Aware Directional Message Passing for Non-Equilibrium Moleculescited by 255
  26. 2020A Note on Over-Smoothing for Graph Neural Networkscited by 249
  27. 2020Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Datacited by 248
  28. 2021Pre-training Molecular Graph Representation with 3D Geometrycited by 248
  29. 2020The Open Catalyst 2020 (OC20) Dataset and Community Challengescited by 246
  30. 2018Graph networks as learnable physics engines for inference and controlcited by 244
  31. 2021Geometric and Physical Quantities Improve E(3) Equivariant Message Passingcited by 228
  32. 2015Semi-supervised Sequence Learningcited by 198
  33. 2020Random Features Strengthen Graph Neural Networkscited by 186
  34. 2021Self-Supervised Learning of Graph Neural Networks: A Unified Reviewcited by 184
  35. 2022TorchMD-NET: Equivariant Transformers for Neural Network based Molecular Potentialscited by 182
  36. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  37. 2021Spherical Message Passing for 3D Graph Networkscited by 168
  38. 2021Learning Gradient Fields for Molecular Conformation Generationcited by 163

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 10; 2023: 34; 2024: 38; 2025: 27; 2026: 15.

124 papers.

Sort:

By area:

  1. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,114
  2. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 703
  3. 2023EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representationscited by 204
  4. 2022A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Languagecited by 131
  5. 2022One Transformer Can Understand Both 2D & 3D Molecular Datacited by 77
  6. 2023Two for One: Diffusion Models and Force Fields for Coarse-Grained Molecular Dynamicscited by 72
  7. 2023GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learningcited by 70
  8. 2022Molecular Geometry Pretraining with SE(3)-Invariant Denoising Distance Matchingcited by 68
  9. 2024Orb: A Fast, Scalable Neural Network Potentialcited by 66
  10. 2023Swallowing the Bitter Pill: Simplified Scalable Conformer Generationcited by 56
  11. 2023Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagationcited by 45
  12. 2023Towards Foundational Models for Molecular Learning on Large-Scale Multi-Task Datasetscited by 39
38 more
  1. 2023From Molecules to Materials: Pre-training Large Generalizable Models for Atomic Property Predictioncited by 39
  2. 2023Simplifying Transformer Blockscited by 37
  3. 2022A Systematic Survey of Chemical Pre-trained Modelscited by 34
  4. 2023Graph Positional and Structural Encodercited by 33
  5. 2023Fractional Denoising for 3D Molecular Pre-trainingcited by 33
  6. 2023A Group Symmetric Stochastic Differential Equation Model for Molecule Multi-modal Pretrainingcited by 32
  7. 2022GPS++: An Optimised Hybrid MPNN/Transformer for Molecular Property Predictioncited by 28
  8. 2024A Survey of Geometric Graph Neural Networks: Data Structures, Models and Applicationscited by 28
  9. 2022A simple, efficient and scalable contrastive masked autoencoder for learning visual representationscited by 25
  10. 2023Evaluation of the MACE Force Field Architecture: from Medicinal Chemistry to Materials Sciencecited by 25
  11. 2023Automated 3D Pre-Training for Molecular Property Predictioncited by 24
  12. 2023Symmetry-Informed Geometric Representation for Molecules, Proteins, and Crystalline Materialscited by 22
  13. 2022Graph Neural Networks for Moleculescited by 21
  14. 2023Multimodal Molecular Pretraining via Modality Blendingcited by 21
  15. 2024UniGEM: A Unified Approach to Generation and Property Prediction for Moleculescited by 21
  16. 2023Generalist Equivariant Transformer Towards 3D Molecular Interaction Learningcited by 19
  17. 2023Sliced Denoising: A Physics-Informed Molecular Pre-Training Methodcited by 18
  18. 2024Generalizing Denoising to Non-Equilibrium Structures Improves Equivariant Force Fieldscited by 18
  19. 2024UniCorn: A Unified Contrastive Learning Approach for Multi-view Molecular Representation Learningcited by 18
  20. 2024Evaluating representation learning on the protein structure universecited by 16
  21. 2024ChemDFM-X: Towards Large Multimodal Model for Chemistrycited by 16
  22. 2025A Periodic Bayesian Flow for Material Generationcited by 16
  23. 2023Protein-ligand binding representation learning from fine-grained interactionscited by 14
  24. 2024Pre-training with Fractional Denoising to Enhance Molecular Property Predictioncited by 14
  25. 2025A Survey of Graph Transformers: Architectures, Theories and Applicationscited by 14
  26. 2023Pre-Training Protein Encoder via Siamese Sequence-Structure Diffusion Trajectory Predictioncited by 13
  27. 2023May the Force be with You: Unified Force-Centric Pre-Training for 3D Molecular Conformationscited by 13
  28. 2023ProFSA: Self-supervised Pocket Pretraining via Protein Fragment-Surroundings Alignmentcited by 13
  29. 2024Geometric Representation Condition Improves Equivariant Molecule Generationcited by 13
  30. 2024Relaxed Equivariance via Multitask Learningcited by 13
  31. 2023Denoise Pretraining on Nonequilibrium Molecules for Accurate and Transferable Neural Potentialscited by 11
  32. 2023Accelerating Molecular Graph Neural Networks via Knowledge Distillationcited by 11
  33. 2024Context-Guided Diffusion for Out-of-Distribution Molecular and Protein Designcited by 11
  34. 2024Towards Dynamic Message Passing on Graphscited by 11
  35. 2023Molecule Joint Auto-Encoding: Trajectory Pretraining with 2D and 3D Diffusioncited by 10
  36. 2025How simple can you go? An off-the-shelf transformer approach to molecular dynamicscited by 10
  37. 2024Molecular relaxation by reverse diffusion with time step predictioncited by 9
  38. 2022Learned Force Fields Are Ready For Ground State Catalyst Discoverycited by 8

50 most cited of 124 All 124 (JSON)