# Transformers for molecular property prediction: Lessons learned from the past five years

arXiv: https://arxiv.org/abs/2404.03969 · HTML: https://mlsanity.com/paper/2404.03969/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 94 references that are in the map.
- First posted on arXiv 5 Apr 2024.

## Builds on (most cited first, 17 of 17)

- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [Analyzing Learned Molecular Representations for Property Prediction](https://mlsanity.com/paper/1904.01561/) (2019)
- [Explainability for Large Language Models: A Survey](https://mlsanity.com/paper/2309.01029/) (2023)
- [Molecular De Novo Design through Deep Reinforcement Learning](https://mlsanity.com/paper/1704.07555/) (2017)
- [Molecular Graph Convolutions: Moving Beyond Fingerprints](https://mlsanity.com/paper/1603.00856/) (2016)
- [Molecular Transformer - A Model for Uncertainty-Calibrated Chemical Reaction Prediction](https://mlsanity.com/paper/1811.02633/) (2018)
- [Large-Scale Chemical Language Representations Capture Molecular Structure and Properties](https://mlsanity.com/paper/2106.09553/) (2021)
- [Learning to Plan Chemical Syntheses](https://mlsanity.com/paper/1708.04202/) (2017)
- [SELFIES and the future of molecular string representations](https://mlsanity.com/paper/2204.00056/) (2022)
- [Regression Transformer: Concurrent sequence regression and generation for molecular language modeling](https://mlsanity.com/paper/2202.01338/) (2022)
- [SELFormer: Molecular Representation Learning via SELFIES Language Models](https://mlsanity.com/paper/2304.04662/) (2023)
- [Artificial Intelligence in Drug Discovery: Applications and Techniques](https://mlsanity.com/paper/2106.05386/) (2021)
- [Validating the Validation: Reanalyzing a large-scale comparison of Deep Learning and Machine Learning models for bioactivity prediction](https://mlsanity.com/paper/1905.11681/) (2019)
- [TwinBooster: Synergising Large Language Models with Barlow Twins and Gradient Boosting for Enhanced Molecular Property Prediction](https://mlsanity.com/paper/2401.04478/) (2024)

## Built on by (most cited first, 8 of 8)

- [A Review of Large Language Models and Autonomous Agents in Chemistry](https://mlsanity.com/paper/2407.01603/) (2024)
- [Synergistic Benefits of Joint Molecule Generation and Property Prediction](https://mlsanity.com/paper/2504.16559/) (2025)
- [Transformers for molecular property prediction: Domain adaptation efficiently improves performance](https://mlsanity.com/paper/2503.03360/) (2025)
- [Advancements in Molecular Property Prediction: A Survey of Single and Multimodal Approaches](https://mlsanity.com/paper/2408.09461/) (2024)
- [MolPILE -- large-scale, diverse dataset for molecular representation learning](https://mlsanity.com/paper/2509.18353/) (2025)
- [TopoFormer: Topology Meets Attention for Graph Learning](https://mlsanity.com/paper/2607.28259/) (2026)
- [Chemical Language Models for Natural Products: A State-Space Model Approach](https://mlsanity.com/paper/2602.13958/) (2026)
- [Does Pretraining Teach Chemical Language Models Chemistry?](https://mlsanity.com/paper/2610.18651/) (2026)

## Abstract (arXiv)

Molecular Property Prediction (MPP) is vital for drug discovery, crop protection, and environmental science. Over the last decades, diverse computational techniques have been developed, from using simple physical and chemical properties and molecular fingerprints in statistical models and classical machine learning to advanced deep learning approaches. In this review, we aim to distill insights from current research on employing transformer models for MPP. We analyze the currently available models and explore key questions that arise when training and fine-tuning a transformer model for MPP. These questions encompass the choice and scale of the pre-training data, optimal architecture selections, and promising pre-training objectives. Our analysis highlights areas not yet covered in current research, inviting further exploration to enhance the field's understanding. Additionally, we address the challenges in comparing different models, emphasizing the need for standardized data splitting and robust statistical analysis.