mlsanityData to 14 Oct 2026

ChemBERTa-2: Towards Chemical Foundation Models

arXiv 2209.01712 ↗First posted 5 Sep 2022cs.LG cs.AI q-bio.BMcited by 127

Not on the map. Why?

Abstract (arXiv)

Large pretrained models such as GPT-3 have had tremendous impact on modern natural language processing by leveraging self-supervised learning to learn salient representations that can be used to readily finetune on a wide variety of downstream tasks. We investigate the possibility of transferring such advances to molecular machine learning by building a chemical foundation model, ChemBERTa-2, using the language of SMILES. While labeled data for molecular prediction tasks is typically scarce, libraries of SMILES strings are readily available. In this work, we build upon ChemBERTa by optimizing the pretraining process. We compare multi-task and self-supervised pretraining by varying hyperparameters and pretraining dataset size, up to 77M compounds from PubChem. To our knowledge, the 77M set constitutes one of the largest datasets used for molecular pretraining to date. We find that with these pretraining improvements, we are competitive with existing state-of-the-art architectures on the MoleculeNet benchmark suite. We analyze the degree to which improvements in pretraining translate to improvement on downstream tasks.

Key facts. Cited by 127 papers in the map (data through 14 Oct 2026). Builds on 13 of its 22 references that are in the map. First posted on arXiv 5 Sep 2022.

Builds on

13 of 22 references.

References · 13

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  5. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
  6. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  7. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,051
  8. 2019Strategies for Pre-training Graph Neural Networkscited by 971
  9. 2019Analyzing Learned Molecular Representations for Property Predictioncited by 375
  10. 2020ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Predictioncited by 337
  11. 2020Molecular representation learning with language models and domain-relevant auxiliary taskscited by 94
  12. 2021Dual-view Molecule Pre-trainingcited by 27
1 more
  1. 2020Learn molecular representations from large-scale unlabeled molecules for drug discoverycited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 19; 2024: 38; 2025: 35; 2026: 31.

127 papers.

Sort:

By area:

  1. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 99
  2. 2023InstructMol: Multi-Modal Integration for Building a Versatile and Reliable Molecular Assistant in Drug Discoverycited by 86
  3. 2023Grammar Prompting for Domain-Specific Language Generation with Large Language Modelscited by 58
  4. 2022polyBERT: A chemical language model to enable fully machine-driven ultrafast polymer informaticscited by 48
  5. 2023SELFormer: Molecular Representation Learning via SELFIES Language Modelscited by 40
  6. 2024Are large language models superhuman chemists?cited by 34
  7. 2024On the Challenges and Opportunities in Generative AIcited by 31
  8. 2022Bidirectional Generation of Structure and Properties Through a Single Molecular Foundation Modelcited by 25
  9. 2024Materials science in the era of large language models: a perspectivecited by 24
  10. 2024Re-Simulation-based Self-Supervised Learning for Pre-Training Foundation Modelscited by 22
  11. 2024textttMiniMol: A Parameter-Efficient Foundation Model for Molecular Learningcited by 20
  12. 2024HANNA: Hard-constraint Neural Network for Consistent Activity Coefficient Predictioncited by 17
38 more
  1. 2025NExT-Mol: 3D Diffusion Meets 1D Language Modeling for 3D Molecule Generationcited by 17
  2. 2023An Interdisciplinary Outlook on Large Language Models for Scientific Researchcited by 15
  3. 2023GPT-MolBERTa: GPT Molecular Features Language Model for molecular property predictioncited by 14
  4. 2025Universally Converging Representations of Matter Across Scientific Foundation Modelscited by 14
  5. 2023SynerGPT: In-Context Learning for Personalized Drug Synergy Prediction and Drug Designcited by 13
  6. 2023Transformers and Large Language Models for Chemistry and Drug Discoverycited by 13
  7. 2024Bio-xLSTM: Generative modeling, representation and in-context learning of biological and chemical sequencescited by 13
  8. 2022MolE: a molecular foundation model for drug discoverycited by 12
  9. 2024MMPolymer: A Multimodal Multitask Pretraining Framework for Polymer Property Predictioncited by 12
  10. 2023MUBen: Benchmarking the Uncertainty of Molecular Representation Modelscited by 11
  11. 2023ADMET property prediction through combinations of molecular fingerprintscited by 11
  12. 2024Less can be more for predicting properties with large language modelscited by 11
  13. 2024Regression with Large Language Models for Materials and Molecular Property Predictioncited by 11
  14. 2024From Generalist to Specialist: A Survey of Large Language Models for Chemistrycited by 11
  15. 2026Structure-Preserving Learning Improves Geometry Generalization in Neural PDEscited by 8
  16. 2024A Survey of Pre-trained Language Models for Processing Scientific Textcited by 7
  17. 2023Transferring a molecular foundation model for polymer property predictionscited by 6
  18. 2024Impact of Domain Knowledge and Multi-Modality on Intelligent Molecular Property Prediction: A Systematic Surveycited by 6
  19. 2024Small Molecule Optimization with Large Language Modelscited by 6
  20. 2025Computational Protein Science in the Era of Large Language Models (LLMs)cited by 6
  21. 2026Agentic reinforcement learning empowers next-generation chemical language models for molecular design and synthesiscited by 6
  22. 2024MolX: Enhancing Large Language Models for Molecular Understanding With A Multi-Modal Extensioncited by 5
  23. 2024A Large Encoder-Decoder Family of Foundation Models For Chemical Languagecited by 5
  24. 2024Tokenization for Molecular Foundation Modelscited by 5
  25. 2025ChEmbed: Enhancing Chemical Literature Search Through Domain-Specific Text Embeddingscited by 5
  26. 2023MHG-GNN: Combination of Molecular Hypergraph Grammar with Graph Neural Networkcited by 4
  27. 2024Structure-Aware E(3)-Invariant Molecular Conformer Aggregation Networkscited by 4
  28. 2024Cross-Modal Learning for Chemistry Property Prediction: Large Language Models Meet Graph Machine Learningcited by 4
  29. 2024A Transformer Based Generative Chemical Language AI Model for Structural Elucidation of Organic Compoundscited by 4
  30. 2024Solaris: A Foundation Model of the Suncited by 4
  31. 2025Transformers for molecular property prediction: Domain adaptation efficiently improves performancecited by 4
  32. 2025High-entropy Advantage in Neural Networks' Generalizabilitycited by 4
  33. 2025oMeBench: Towards Robust Benchmarking of LLMs in Organic Mechanism Elucidation and Reasoningcited by 4
  34. 2024Improving Legal Judgement Prediction in Romanian with Long Text Encoderscited by 3
  35. 2024Explainable Molecular Property Prediction: Aligning Chemical Concepts with Predictions via Language Modelscited by 3
  36. 2024FusionDTI: Fine-grained Binding Discovery with Token-level Fusion for Drug-Target Interactioncited by 3
  37. 2024Leveraging Chemistry Foundation Models to Facilitate Structure Focused Retrieval Augmented Generation in Multi-Agent Workflows for Catalyst and Materials Designcited by 3
  38. 2024Property Enhanced Instruction Tuning for Multi-task Molecule Generation with Large Language Modelscited by 3

50 most cited of 127 All 127 (JSON)