# Transferring a molecular foundation model for polymer property predictions

arXiv: https://arxiv.org/abs/2310.16958 · HTML: https://mlsanity.com/paper/2310.16958/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 46 references that are in the map.
- First posted on arXiv 25 Oct 2023.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](https://mlsanity.com/paper/1904.00962/) (2019)
- [Large-Scale Chemical Language Representations Capture Molecular Structure and Properties](https://mlsanity.com/paper/2106.09553/) (2021)
- [ChemBERTa-2: Towards Chemical Foundation Models](https://mlsanity.com/paper/2209.01712/) (2022)
- [polyBERT: A chemical language model to enable fully machine-driven ultrafast polymer informatics](https://mlsanity.com/paper/2209.14803/) (2022)
- [TransPolymer: a Transformer-based language model for polymer property predictions](https://mlsanity.com/paper/2209.01307/) (2022)
- [Polymer Informatics with Multi-Task Learning](https://mlsanity.com/paper/2010.15166/) (2020)
- [Polymer Informatics: Current Status and Critical Next Steps](https://mlsanity.com/paper/2011.00508/) (2020)
- [Copolymer Informatics with Multi-Task Deep Neural Networks](https://mlsanity.com/paper/2103.14174/) (2021)
- [Bioplastic Design using Multitask Deep Neural Networks](https://mlsanity.com/paper/2203.12033/) (2022)

## Built on by (most cited first, 6 of 6)

- [MMPolymer: A Multimodal Multitask Pretraining Framework for Polymer Property Prediction](https://mlsanity.com/paper/2406.04727/) (2024)
- [Unified Cross-Scale 3D Generation and Understanding via Autoregressive Modeling](https://mlsanity.com/paper/2503.16278/) (2025)
- [Multimodal machine learning with large language embedding model for polymer property prediction](https://mlsanity.com/paper/2503.22962/) (2025)
- [Joint Embedding Predictive Architecture for self-supervised pretraining on polymer molecular graphs](https://mlsanity.com/paper/2506.18194/) (2025)
- [MIPS: a Multimodal Infinite Polymer Sequence Pre-training Framework for Polymer Property Prediction](https://mlsanity.com/paper/2507.20326/) (2025)
- [A Conformation-Centric Generative Foundation Model for Linear Polymer Modeling and Design](https://mlsanity.com/paper/2510.16023/) (2025)

## Abstract (arXiv)

Transformer-based large language models have remarkable potential to accelerate design optimization for applications such as drug development and materials discovery. Self-supervised pretraining of transformer models requires large-scale datasets, which are often sparsely populated in topical areas such as polymer science. State-of-the-art approaches for polymers conduct data augmentation to generate additional samples but unavoidably incurs extra computational costs. In contrast, large-scale open-source datasets are available for small molecules and provide a potential solution to data scarcity through transfer learning. In this work, we show that using transformers pretrained on small molecules and fine-tuned on polymer properties achieve comparable accuracy to those trained on augmented polymer datasets for a series of benchmark prediction tasks.