mlsanityData to 14 Oct 2026

Does Pretraining Teach Chemical Language Models Chemistry?

arXiv 2610.18651 ↗First posted 13 Oct 2026cs.LG cs.CL physics.chem-phcited by 0

Not on the map. Why?

Abstract (arXiv)

Chemical language models (CLMs) are pretrained on large corpora of molecules with the aim of learning representations that transfer to diverse downstream tasks. Here, we systematically study the effects of pretraining using a series of such CLMs across 64 molecular property prediction datasets. We find that the downstream gains of pretraining in CLMs do not scale and performance saturates below gradient-boosting baselines. Across the regimes evaluated, only supervised pretraining on traditional physico-chemical descriptors consistently improves downstream performance for the CLMs tested whereas some forms of pretraining result in models worse than a randomly initialized encoder. While models can resolve molecular structure well, they do so no better than a randomly initialized model, suggesting that, at best, CLMs learn descriptor-like properties in addition to low-level token statistics during pretraining, but surprisingly little chemistry beyond what classical molecular features already capture.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 28 of its 51 references that are in the map. First posted on arXiv 13 Oct 2026.

Builds on

28 of 51 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  3. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  5. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  6. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,051
  7. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  8. 2015Convolutional Networks on Graphs for Learning Molecular Fingerprintscited by 754
  9. 2018Do Better ImageNet Models Transfer Better?cited by 565
  10. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 383
  11. 2019Transfusion: Understanding Transfer Learning for Medical Imagingcited by 368
  12. 2021Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Developmentcited by 264
16 more
  1. 2018Molecular Transformer - A Model for Uncertainty-Calibrated Chemical Reaction Predictioncited by 229
  2. 2021Large-Scale Chemical Language Representations Capture Molecular Structure and Propertiescited by 206
  3. 2020Molecular representation learning with language models and domain-relevant auxiliary taskscited by 94
  4. 2022Does GNN Pretraining Help Molecular Representation?cited by 60
  5. 2017SMILES Enumeration as Data Augmentation for Neural Network Modeling of Moleculescited by 59
  6. 2023Towards Foundational Models for Molecular Learning on Large-Scale Multi-Task Datasetscited by 39
  7. 2022Insights into Pre-training via Simpler Synthetic Taskscited by 26
  8. 2024textttMiniMol: A Parameter-Efficient Foundation Model for Molecular Learningcited by 20
  9. 2024Transformers for molecular property prediction: Lessons learned from the past five yearscited by 8
  10. 2024Tokenization for Molecular Foundation Modelscited by 5
  11. 2024SELF-BART : A Transformer-based Molecular Representation Model using SELFIEScited by 4
  12. 2025Transformers for molecular property prediction: Domain adaptation efficiently improves performancecited by 4
  13. 2026What Does a Chemical Language Model Know About Molecules?cited by 2
  14. 2025SmilesT5: Domain-specific pretraining for molecular language modelscited by 1
  15. 2026How Well Do Large-Scale Chemical Language Models Transfer to Downstream Tasks?cited by 1
  16. 2026Probing Chemical Language Models: Effects of Pre-training and Fine-tuningcited by 1

Built on by

Not cited yet.