mlsanityData to 7 Oct 2026

ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction

arXiv 2010.09885 ↗First posted 19 Oct 2020cs.LG cs.CL physics.chem-phcited by 332

Not on the map. Why?

Abstract (arXiv)

GNNs and chemical fingerprints are the predominant approaches to representing molecules for property prediction. However, in NLP, transformers have become the de-facto standard for representation learning thanks to their strong downstream task transfer. In parallel, the software ecosystem around transformers is maturing rapidly, with libraries like HuggingFace and BertViz enabling streamlined training and introspection. In this work, we make one of the first attempts to systematically evaluate transformers on molecular property prediction tasks via our ChemBERTa model. ChemBERTa scales well with pretraining dataset size, offering competitive downstream performance on MoleculeNet and useful attention-based visualization modalities. Our results suggest that transformers offer a promising avenue of future work for molecular representation learning and property prediction. To facilitate these efforts, we release a curated dataset of 77M SMILES from PubChem suitable for large-scale self-supervised pretraining.

Key facts. Cited by 332 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 17; 2022: 30; 2023: 50; 2024: 72; 2025: 83; 2026: 80.

332 papers.

Sort:

By area:

  1. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 609
  2. 2021Large-Scale Chemical Language Representations Capture Molecular Structure and Propertiescited by 203
  3. 2022Translation between Molecules and Natural Languagecited by 173
  4. 2022A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Languagecited by 129
  5. 2022ChemBERTa-2: Towards Chemical Foundation Modelscited by 123
  6. 2021Geometric Deep Learning on Molecular Representationscited by 113
  7. 2023BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associationscited by 100
  8. 2022Accelerating Bayesian Optimization for Biological Sequence Design with Denoising Autoencoderscited by 99
  9. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  10. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 96
  11. 2023Empowering Molecule Discovery for Molecule-Caption Translation with Large Language Models: A ChatGPT Perspectivecited by 91
  12. 2023Multiple Physics Pretraining for Physical Surrogate Modelscited by 90
38 more
  1. 2023InstructMol: Multi-Modal Integration for Building a Versatile and Reliable Molecular Assistant in Drug Discoverycited by 86
  2. 2024LLM and Simulation as Bilevel Optimizers: A New Paradigm to Advance Physical Scientific Discoverycited by 82
  3. 2024Scientific Large Language Models: A Survey on Biological & Chemical Domainscited by 80
  4. 2023GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learningcited by 70
  5. 2022MolGenSurvey: A Systematic Survey in Machine Learning Models for Molecule Designcited by 66
  6. 2024Efficient Evolutionary Search Over Chemical Space with Large Language Modelscited by 59
  7. 2024BioT5+: Towards Generalized Biological Understanding with IUPAC Integration and Multi-task Tuningcited by 54
  8. 2023MolFM: A Multimodal Molecular Foundation Modelcited by 50
  9. 2022polyBERT: A chemical language model to enable fully machine-driven ultrafast polymer informaticscited by 48
  10. 2024A Sober Look at LLMs for Material Discovery: Are They Actually Good for Bayesian Optimization Over Molecules?cited by 46
  11. 2022Unified 2D and 3D Pre-Training of Molecular Representationscited by 45
  12. 2021Molformer: Motif-based Transformer on 3D Heterogeneous Molecular Graphscited by 44
  13. 2022TransPolymer: a Transformer-based language model for polymer property predictionscited by 43
  14. 2023Can Large Language Models Empower Molecular Property Prediction?cited by 43
  15. 2023SELFormer: Molecular Representation Learning via SELFIES Language Modelscited by 40
  16. 2024DrugAgent: Automating AI-aided Drug Discovery Programming through LLM Multi-Agent Collaborationcited by 40
  17. 2022Improving Molecular Contrastive Learning via Faulty Negative Mitigation and Decomposed Fragment Contrastcited by 37
  18. 2021Chemical-Reaction-Aware Molecule Representation Learningcited by 36
  19. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
  20. 2022Crystal Twins: Self-supervised Learning for Crystalline Material Property Predictioncited by 34
  21. 2023Fractional Denoising for 3D Molecular Pre-trainingcited by 33
  22. 2022Group SELFIES: A Robust Fragment-Based Molecular String Representationcited by 32
  23. 2023Prot2Text: Multimodal Protein's Function Generation with GNNs and Transformerscited by 32
  24. 2024Graph Transformers: A Surveycited by 32
  25. 2021Artificial Intelligence in Drug Discovery: Applications and Techniquescited by 28
  26. 2021Dual-view Molecule Pre-trainingcited by 27
  27. 2025Walrus: A Cross-Domain Foundation Model for Continuum Dynamicscited by 26
  28. 2023ReLM: Leveraging Language Models for Enhanced Chemical Reaction Predictioncited by 24
  29. 2024Large Language Models for Manufacturingcited by 24
  30. 2023Uncovering Neural Scaling Laws in Molecular Representation Learningcited by 23
  31. 2024Materials science in the era of large language models: a perspectivecited by 23
  32. 2024UniMoT: Unified Molecule-Text Language Model with Discrete Token Representationcited by 23
  33. 2021Relative Molecule Self-Attention Transformercited by 21
  34. 2023Multimodal Molecular Pretraining via Modality Blendingcited by 21
  35. 2022A graph representation of molecular ensembles for polymer property predictioncited by 20
  36. 2025OmniScience: A Domain-Specialized LLM for Scientific Reasoning and Discoverycited by 20
  37. 2024ReactXT: Understanding Molecular "Reaction-ship" via Reaction-Contextualized Molecule-Text Pretrainingcited by 19
  38. 2022BARTSmiles: Generative Masked Language Models for Molecular Representationscited by 18

50 most cited of 332 All 332 (JSON)