mlsanityData to 14 Oct 2026

Uncovering Neural Scaling Laws in Molecular Representation Learning

arXiv 2309.15123 ↗First posted 15 Sep 2023physics.chem-ph cs.LGcited by 23

Not on the map. Why?

Abstract (arXiv)

Molecular Representation Learning (MRL) has emerged as a powerful tool for drug and materials discovery in a variety of tasks such as virtual screening and inverse design. While there has been a surge of interest in advancing model-centric techniques, the influence of both data quantity and quality on molecular representations is not yet clearly understood within this field. In this paper, we delve into the neural scaling behaviors of MRL from a data-centric viewpoint, examining four key dimensions: (1) data modalities, (2) dataset splitting, (3) the role of pre-training, and (4) model capacity. Our empirical studies confirm a consistent power-law relationship between data volume and MRL performance across these dimensions. Additionally, through detailed analysis, we identify potential avenues for improving learning efficiency. To challenge these scaling laws, we adapt seven popular data pruning strategies to molecular data and benchmark their performance. Our findings underline the importance of data-centric MRL and highlight possible directions for future research.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 52 of its 86 references that are in the map. First posted on arXiv 15 Sep 2023.

Builds on

52 of 86 references.

References · 52

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2018How Powerful are Graph Neural Networks?cited by 4,393
  6. 2017Neural Message Passing for Quantum Chemistrycited by 3,487
  7. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,434
  8. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  9. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,163
  10. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  11. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,051
  12. 2019Strategies for Pre-training Graph Neural Networkscited by 971
38 more
  1. 2021E(n) Equivariant Graph Neural Networkscited by 965
  2. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  3. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
  4. 2020SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networkscited by 607
  5. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  6. 2017SchNet: A continuous-filter convolutional neural network for modeling quantum interactionscited by 571
  7. 2020Directional Message Passing for Molecular Graphscited by 561
  8. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
  9. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  10. 2020Principal Neighbourhood Aggregation for Graph Netscited by 478
  11. 2022MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fieldscited by 474
  12. 2022GraphMAE: Self-Supervised Masked Graph Autoencoderscited by 448
  13. 2021Equivariant message passing for the prediction of tensorial properties and molecular spectracited by 441
  14. 2021GemNet: Universal Directional Graph Neural Networks for Moleculescited by 343
  15. 2020ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Predictioncited by 337
  16. 2020Improving Graph Neural Network Expressivity via Subgraph Isomorphism Countingcited by 313
  17. 2021Scaling Laws for Transfercited by 272
  18. 2022Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphscited by 264
  19. 2020DeeperGCN: All You Need to Train Deeper GCNscited by 250
  20. 2021Pre-training Molecular Graph Representation with 3D Geometrycited by 248
  21. 2021Weisfeiler and Lehman Go Cellular: CW Networkscited by 218
  22. 2021Large-Scale Chemical Language Representations Capture Molecular Structure and Propertiescited by 206
  23. 2021ChemRL-GEM: Geometry Enhanced Molecular Representation Learning for Property Predictioncited by 204
  24. 2022e3nn: Euclidean Neural Networkscited by 174
  25. 2022DeepCore: A Comprehensive Library for Coreset Selection in Deep Learningcited by 147
  26. 2020Directional Graph Networkscited by 133
  27. 2023Artificial Intelligence for Science in Quantum, Atomistic, and Continuum Systemscited by 121
  28. 2022Coverage-centric Coreset Selection for High Pruning Ratescited by 117
  29. 2022Broken Neural Scaling Lawscited by 109
  30. 2021SE(3) Equivariant Graph Neural Networks with Complete Local Framescited by 76
  31. 2022MolGenSurvey: A Systematic Survey in Machine Learning Models for Molecule Designcited by 66
  32. 2021GeomGCL: Geometric Graph Contrastive Learning for Molecular Property Predictioncited by 61
  33. 2021Equivariant Graph Neural Networks for 3D Macromolecular Structurecited by 53
  34. 2023A new perspective on building efficient and expressive 3D equivariant graph neural networkscited by 47
  35. 2022Unified 2D and 3D Pre-Training of Molecular Representationscited by 45
  36. 2022Scaling Laws for a Multi-Agent Reinforcement Learning Modelcited by 38
  37. 2022A Systematic Survey of Chemical Pre-trained Modelscited by 34
  38. 2022Molecular Representation Learning via Heterogeneous Motif Graph Neural Networkscited by 23

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 13; 2025: 8; 2026: 2.

23 papers.

Sort:

By area:

  1. 2024Position: Graph Foundation Models are Already Herecited by 86
  2. 2024On the Scalability of GNNs for Molecular Graphscited by 36
  3. 2024GLBench: A Comprehensive Benchmark for Graph with Large Language Modelscited by 36
  4. 2024Text-Guided Molecule Generation with Diffusion Language Modelcited by 31
  5. 2024Uni-Mol2: Exploring Molecular Pretraining Model at Scalecited by 20
  6. 2024Towards Neural Scaling Laws on Graphscited by 13
  7. 2024Rethinking Fair Graph Neural Networks from Re-balancingcited by 11
  8. 2025Materials Generation in the Era of Artificial Intelligence: A Comprehensive Surveycited by 10
  9. 2024Beyond Efficiency: Molecular Data Pruning for Enhanced Generalizationcited by 9
  10. 2024GDeR: Safeguarding Efficiency, Balancing, and Robustness via Prototypical Graph Pruningcited by 8
  11. 2025MolSpectra: Pre-training 3D Molecular Representation with Multi-modal Energy Spectracited by 8
  12. 2025The quest for the GRAph Level autoEncoder (GRALE)cited by 8
11 more
  1. 2024Pin-Tuning: Parameter-Efficient In-Context Tuning for Few-Shot Molecular Property Predictioncited by 4
  2. 2024Benchmarking Out-of-Distribution Generalization Capabilities of DNN-based Encoding Models for the Ventral Visual Cortexcited by 3
  3. 2024GNN-SKAN: Harnessing the Power of SwallowKAN to Advance Molecular Representation Learning with GNNscited by 3
  4. 2024Pushing the Limits of All-Atom Geometric Graph Neural Networks: Pre-Training, Scaling and Zero-Shot Transfercited by 3
  5. 2025CLOUD: A Scalable and Physics-Informed Foundation Model for Crystal Representation Learningcited by 3
  6. 2025Representational Alignment with Chemical Induced Fit for Molecular Relational Learningcited by 2
  7. 2026OrderDP: A Theoretically Guaranteed Lossless Dynamic Data Pruning Frameworkcited by 2
  8. 2025Graffe: Graph Representation Learning via Diffusion Probabilistic Modelscited by 1
  9. 2026How Well Do Large-Scale Chemical Language Models Transfer to Downstream Tasks?cited by 1
  10. 2025Cross-Modality Controlled Molecule Generation with Diffusion Language Modelcited by 0
  11. 2025M-GLC: Motif-Driven Global-Local Context Graphs for Few-shot Molecular Property Predictioncited by 0