# TransPolymer: a Transformer-based language model for polymer property predictions

arXiv: https://arxiv.org/abs/2209.01307 · HTML: https://mlsanity.com/paper/2209.01307/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 78 references that are in the map.
- First posted on arXiv 3 Sep 2022.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [Convolutional Networks on Graphs for Learning Molecular Fingerprints](https://mlsanity.com/paper/1509.09292/) (2015)
- [Analyzing Learned Molecular Representations for Property Prediction](https://mlsanity.com/paper/1904.01561/) (2019)
- [ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction](https://mlsanity.com/paper/2010.09885/) (2020)
- [Revisiting Few-sample BERT Fine-tuning](https://mlsanity.com/paper/2006.05987/) (2020)
- [Molecular Transformer - A Model for Uncertainty-Calibrated Chemical Reaction Prediction](https://mlsanity.com/paper/1811.02633/) (2018)
- [UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training](https://mlsanity.com/paper/2002.12804/) (2020)
- [Dissecting Contextual Word Embeddings: Architecture and Representation](https://mlsanity.com/paper/1808.08949/) (2018)
- [Self-Attention Attribution: Interpreting Information Interactions Inside Transformer](https://mlsanity.com/paper/2004.11207/) (2020)
- [SMILES Transformer: Pre-trained Molecular Fingerprint for Low Data Drug Discovery](https://mlsanity.com/paper/1911.04738/) (2019)
- [Keeping it Simple: Language Models can learn Complex Molecular Distributions](https://mlsanity.com/paper/2112.03041/) (2021)
- ["Found in Translation": Predicting Outcomes of Complex Organic Chemistry Reactions using Neural Sequence-to-Sequence Models](https://mlsanity.com/paper/1711.04810/) (2017)
- [Universal Sentence Representation Learning with Conditional Masked Language Model](https://mlsanity.com/paper/2012.14388/) (2020)
- [Crystal Twins: Self-supervised Learning for Crystalline Material Property Prediction](https://mlsanity.com/paper/2205.01893/) (2022)
- [Message-passing neural networks for high-throughput polymer screening](https://mlsanity.com/paper/1807.10363/) (2018)
- [Orbital Graph Convolutional Neural Network for Material Property Prediction](https://mlsanity.com/paper/2008.06415/) (2020)
- [Polymer Informatics with Multi-Task Learning](https://mlsanity.com/paper/2010.15166/) (2020)
- [Polymer Informatics: Current Status and Critical Next Steps](https://mlsanity.com/paper/2011.00508/) (2020)
- [A graph representation of molecular ensembles for polymer property prediction](https://mlsanity.com/paper/2205.08619/) (2022)
- [Polymers for Extreme Conditions Designed Using Syntax-Directed Variational Autoencoders](https://mlsanity.com/paper/2011.02551/) (2020)
- [Accelerating amorphous polymer electrolyte screening by learning to reduce errors in molecular dynamics simulated properties](https://mlsanity.com/paper/2101.05339/) (2021)
- [SMILES-X: autonomous molecular compounds characterization for small datasets without descriptors](https://mlsanity.com/paper/1906.09938/) (2019)

## Built on by (most cited first, 30 of 43)

- [A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery](https://mlsanity.com/paper/2406.10833/) (2024)
- [polyBERT: A chemical language model to enable fully machine-driven ultrafast polymer informatics](https://mlsanity.com/paper/2209.14803/) (2022)
- [MOFormer: Self-Supervised Transformer model for Metal-Organic Framework Property Prediction](https://mlsanity.com/paper/2210.14188/) (2022)
- [Bidirectional Generation of Structure and Properties Through a Single Molecular Foundation Model](https://mlsanity.com/paper/2211.10590/) (2022)
- [Bayesian Optimization of Catalysis With In-Context Learning](https://mlsanity.com/paper/2304.05341/) (2023)
- [GPT-MolBERTa: GPT Molecular Features Language Model for molecular property prediction](https://mlsanity.com/paper/2310.03030/) (2023)
- [Reinforcement Feature Transformation for Polymer Property Performance Prediction](https://mlsanity.com/paper/2409.15616/) (2024)
- [MMPolymer: A Multimodal Multitask Pretraining Framework for Polymer Property Prediction](https://mlsanity.com/paper/2406.04727/) (2024)
- [Evaluating the Performance and Robustness of LLMs in Materials Science Q&A and Property Predictions](https://mlsanity.com/paper/2409.14572/) (2024)
- [Unified Cross-Scale 3D Generation and Understanding via Autoregressive Modeling](https://mlsanity.com/paper/2503.16278/) (2025)
- [Materials Informatics Transformer: A Language Model for Interpretable Materials Properties Prediction](https://mlsanity.com/paper/2308.16259/) (2023)
- [PolyCL: Contrastive Learning for Polymer Representation Learning via Explicit and Implicit Augmentations](https://mlsanity.com/paper/2408.07556/) (2024)
- [Inverse Design of Copolymers Including Stoichiometry and Chain Architecture](https://mlsanity.com/paper/2410.02824/) (2024)
- [Transferring a molecular foundation model for polymer property predictions](https://mlsanity.com/paper/2310.16958/) (2023)
- [Tabular foundation models for in-context prediction of molecular properties](https://mlsanity.com/paper/2604.16123/) (2026)
- [AlloyBERT: Alloy Property Prediction with Large Language Models](https://mlsanity.com/paper/2403.19783/) (2024)
- [Tokenization for Molecular Foundation Models](https://mlsanity.com/paper/2409.15370/) (2024)
- [Multimodal machine learning with large language embedding model for polymer property prediction](https://mlsanity.com/paper/2503.22962/) (2025)
- [Artificial Intelligence in Materials Science and Engineering: Current Landscape, Key Challenges, and Future Trajectorie](https://mlsanity.com/paper/2601.12554/) (2026)
- [High-entropy Advantage in Neural Networks' Generalizability](https://mlsanity.com/paper/2503.13145/) (2025)
- [Leveraging Chemistry Foundation Models to Facilitate Structure Focused Retrieval Augmented Generation in Multi-Agent Workflows for Catalyst and Materials Design](https://mlsanity.com/paper/2408.11793/) (2024)
- [PolyConf: Unlocking Polymer Conformation Generation through Hierarchical Generative Models](https://mlsanity.com/paper/2504.08859/) (2025)
- [Polymer-Agent: Large Language Model Agent for Polymer Design](https://mlsanity.com/paper/2601.16376/) (2026)
- [Large language models, physics-based modeling, experimental measurements: the trinity of data-scarce learning of polymer properties](https://mlsanity.com/paper/2407.02770/) (2024)
- [Molecular topological deep learning for polymer property prediction](https://mlsanity.com/paper/2410.04765/) (2024)
- [Predicting CO2 Absorption in Ionic Liquids with Molecular Descriptors and Explainable Graph Neural Networks](https://mlsanity.com/paper/2210.01120/) (2022)
- [MolTRES: Improving Chemical Language Representation Learning for Molecular Property Prediction](https://mlsanity.com/paper/2408.01426/) (2024)
- [Joint Embedding Predictive Architecture for self-supervised pretraining on polymer molecular graphs](https://mlsanity.com/paper/2506.18194/) (2025)
- [From Natural Language to Materials Discovery:The Materials Knowledge Navigation Agent](https://mlsanity.com/paper/2602.11123/) (2026)
- [Controllable Molecular Generative Foundation Models](https://mlsanity.com/paper/2605.15354/) (2026)

## Abstract (arXiv)

Accurate and efficient prediction of polymer properties is of great significance in polymer design. Conventionally, expensive and time-consuming experiments or simulations are required to evaluate polymer functions. Recently, Transformer models, equipped with self-attention mechanisms, have exhibited superior performance in natural language processing. However, such methods have not been investigated in polymer sciences. Herein, we report TransPolymer, a Transformer-based language model for polymer property prediction. Our proposed polymer tokenizer with chemical awareness enables learning representations from polymer sequences. Rigorous experiments on ten polymer property prediction benchmarks demonstrate the superior performance of TransPolymer. Moreover, we show that TransPolymer benefits from pretraining on large unlabeled dataset via Masked Language Modeling. Experimental results further manifest the important role of self-attention in modeling polymer sequences. We highlight this model as a promising computational tool for promoting rational polymer design and understanding structure-property relationships from a data science view.