# Ensemble Model With Bert,Roberta and Xlnet For Molecular property prediction

arXiv: https://arxiv.org/abs/2406.06553 · HTML: https://mlsanity.com/paper/2406.06553/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 67 references that are in the map.
- First posted on arXiv 30 May 2024.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [Automatic chemical design using a data-driven continuous representation of molecules](https://mlsanity.com/paper/1610.02415/) (2016)
- [Convolutional Networks on Graphs for Learning Molecular Fingerprints](https://mlsanity.com/paper/1509.09292/) (2015)
- [Analyzing Learned Molecular Representations for Property Prediction](https://mlsanity.com/paper/1904.01561/) (2019)
- [ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction](https://mlsanity.com/paper/2010.09885/) (2020)
- [Large-Scale Chemical Language Representations Capture Molecular Structure and Properties](https://mlsanity.com/paper/2106.09553/) (2021)
- [Motif-based Graph Self-Supervised Learning for Molecular Property Prediction](https://mlsanity.com/paper/2110.00987/) (2021)
- [Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence](https://mlsanity.com/paper/2004.03974/) (2020)
- [Exploring the Limits of Large Scale Pre-training](https://mlsanity.com/paper/2110.02095/) (2021)
- [Molecular representation learning with language models and domain-relevant auxiliary tasks](https://mlsanity.com/paper/2011.13230/) (2020)
- [SMILES Transformer: Pre-trained Molecular Fingerprint for Low Data Drug Discovery](https://mlsanity.com/paper/1911.04738/) (2019)
- [How Useful is Self-Supervised Pretraining for Visual Tasks?](https://mlsanity.com/paper/2003.14323/) (2020)
- [Regression Transformer: Concurrent sequence regression and generation for molecular language modeling](https://mlsanity.com/paper/2202.01338/) (2022)
- [Conditional molecular design with deep generative models](https://mlsanity.com/paper/1805.00108/) (2018)
- [What the [MASK]? Making Sense of Language-Specific BERT Models](https://mlsanity.com/paper/2003.02912/) (2020)
- [Deep Learning for Computational Chemistry](https://mlsanity.com/paper/1701.04503/) (2017)
- [SMILES2Vec: An Interpretable General-Purpose Deep Neural Network for Predicting Chemical Properties](https://mlsanity.com/paper/1712.02034/) (2017)
- [All SMILES Variational Autoencoder](https://mlsanity.com/paper/1905.13343/) (2019)

## Abstract (arXiv)

This paper presents a novel approach for predicting molecular properties with high accuracy without the need for extensive pre-training. Employing ensemble learning and supervised fine-tuning of BERT, RoBERTa, and XLNet, our method demonstrates significant effectiveness compared to existing advanced models. Crucially, it addresses the issue of limited computational resources faced by experimental groups, enabling them to accurately predict molecular properties. This innovation provides a cost-effective and resource-efficient solution, potentially advancing further research in the molecular domain.