# polyBERT: A chemical language model to enable fully machine-driven ultrafast polymer informatics

arXiv: https://arxiv.org/abs/2209.14803 · HTML: https://mlsanity.com/paper/2209.14803/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 73 references that are in the map.
- First posted on arXiv 29 Sep 2022.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [Quantifying the Carbon Emissions of Machine Learning](https://mlsanity.com/paper/1910.09700/) (2019)
- [ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction](https://mlsanity.com/paper/2010.09885/) (2020)
- [A Multiscale Visualization of Attention in the Transformer Model](https://mlsanity.com/paper/1906.05714/) (2019)
- [Molecular Transformer - A Model for Uncertainty-Calibrated Chemical Reaction Prediction](https://mlsanity.com/paper/1811.02633/) (2018)
- [ChemRL-GEM: Geometry Enhanced Molecular Representation Learning for Property Prediction](https://mlsanity.com/paper/2106.06130/) (2021)
- [ChemBERTa-2: Towards Chemical Foundation Models](https://mlsanity.com/paper/2209.01712/) (2022)
- [The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?](https://mlsanity.com/paper/2010.05607/) (2020)
- [TransPolymer: a Transformer-based language model for polymer property predictions](https://mlsanity.com/paper/2209.01307/) (2022)
- [Polymer Informatics with Multi-Task Learning](https://mlsanity.com/paper/2010.15166/) (2020)
- [Polymer Informatics: Current Status and Critical Next Steps](https://mlsanity.com/paper/2011.00508/) (2020)
- [A graph representation of molecular ensembles for polymer property prediction](https://mlsanity.com/paper/2205.08619/) (2022)
- [Polymers for Extreme Conditions Designed Using Syntax-Directed Variational Autoencoders](https://mlsanity.com/paper/2011.02551/) (2020)
- [Copolymer Informatics with Multi-Task Deep Neural Networks](https://mlsanity.com/paper/2103.14174/) (2021)
- [Bioplastic Design using Multitask Deep Neural Networks](https://mlsanity.com/paper/2203.12033/) (2022)

## Built on by (most cited first, 30 of 48)

- [Chatlaw: A Multi-Agent Legal Assistant based on a Role-Aligned Mixture-of-Experts Architecture](https://mlsanity.com/paper/2306.16092/) (2023)
- [A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery](https://mlsanity.com/paper/2406.10833/) (2024)
- [Bidirectional Generation of Structure and Properties Through a Single Molecular Foundation Model](https://mlsanity.com/paper/2211.10590/) (2022)
- [Offline Model-Based Optimization: Comprehensive Review](https://mlsanity.com/paper/2503.17286/) (2025)
- [MMPolymer: A Multimodal Multitask Pretraining Framework for Polymer Property Prediction](https://mlsanity.com/paper/2406.04727/) (2024)
- [Scalable Training of Trustworthy and Energy-Efficient Predictive Graph Foundation Models for Atomistic Materials Modeling: A Case Study with HydraGNN](https://mlsanity.com/paper/2406.12909/) (2024)
- [Unified Cross-Scale 3D Generation and Understanding via Autoregressive Modeling](https://mlsanity.com/paper/2503.16278/) (2025)
- [MAGE: Model-Level Graph Neural Networks Explanations via Motif-based Graph Generation](https://mlsanity.com/paper/2405.12519/) (2024)
- [PolyCL: Contrastive Learning for Polymer Representation Learning via Explicit and Implicit Augmentations](https://mlsanity.com/paper/2408.07556/) (2024)
- [Inverse Design of Copolymers Including Stoichiometry and Chain Architecture](https://mlsanity.com/paper/2410.02824/) (2024)
- [Transferring a molecular foundation model for polymer property predictions](https://mlsanity.com/paper/2310.16958/) (2023)
- [Tabular foundation models for in-context prediction of molecular properties](https://mlsanity.com/paper/2604.16123/) (2026)
- [Multimodal machine learning with large language embedding model for polymer property prediction](https://mlsanity.com/paper/2503.22962/) (2025)
- [Artificial Intelligence in Materials Science and Engineering: Current Landscape, Key Challenges, and Future Trajectorie](https://mlsanity.com/paper/2601.12554/) (2026)
- [Bridging AI and Science: Implications from a Large-Scale Literature Analysis of AI4Science](https://mlsanity.com/paper/2412.09628/) (2024)
- [PolyConf: Unlocking Polymer Conformation Generation through Hierarchical Generative Models](https://mlsanity.com/paper/2504.08859/) (2025)
- [Polymer-Agent: Large Language Model Agent for Polymer Design](https://mlsanity.com/paper/2601.16376/) (2026)
- [Large language models, physics-based modeling, experimental measurements: the trinity of data-scarce learning of polymer properties](https://mlsanity.com/paper/2407.02770/) (2024)
- [Machine Learning Based Prediction of Proton Conductivity in Metal-Organic Frameworks](https://mlsanity.com/paper/2407.09514/) (2024)
- [Molecular topological deep learning for polymer property prediction](https://mlsanity.com/paper/2410.04765/) (2024)
- [What Does a Chemical Language Model Know About Molecules?](https://mlsanity.com/paper/2606.23443/) (2026)
- [MotifPiece: A Data-Driven Approach for Effective Motif Extraction and Molecular Representation Learning](https://mlsanity.com/paper/2312.15387/) (2023)
- [Accelerating materials discovery for polymer solar cells: Data-driven insights enabled by natural language processing](https://mlsanity.com/paper/2402.19462/) (2024)
- [MolTRES: Improving Chemical Language Representation Learning for Molecular Property Prediction](https://mlsanity.com/paper/2408.01426/) (2024)
- [Joint Embedding Predictive Architecture for self-supervised pretraining on polymer molecular graphs](https://mlsanity.com/paper/2506.18194/) (2025)
- [Open Polymer Challenge: Post-Competition Report](https://mlsanity.com/paper/2512.08896/) (2025)
- [Topology-Aware Multiscale Mixture of Experts for Efficient Molecular Property Prediction](https://mlsanity.com/paper/2601.12637/) (2026)
- [MoleCode unlocks structural intelligence in large language models](https://mlsanity.com/paper/2605.16480/) (2026)
- [A Multi-Agent Framework for Automated Coarse-Grained Molecular Dynamics of Polymers](https://mlsanity.com/paper/2608.06694/) (2026)
- [AI-Driven Discovery of High Performance Polymer Electrodes for Next-Generation Batteries](https://mlsanity.com/paper/2502.13899/) (2025)

## Abstract (arXiv)

Polymers are a vital part of everyday life. Their chemical universe is so large that it presents unprecedented opportunities as well as significant challenges to identify suitable application-specific candidates. We present a complete end-to-end machine-driven polymer informatics pipeline that can search this space for suitable candidates at unprecedented speed and accuracy. This pipeline includes a polymer chemical fingerprinting capability called polyBERT (inspired by Natural Language Processing concepts), and a multitask learning approach that maps the polyBERT fingerprints to a host of properties. polyBERT is a chemical linguist that treats the chemical structure of polymers as a chemical language. The present approach outstrips the best presently available concepts for polymer property prediction based on handcrafted fingerprint schemes in speed by two orders of magnitude while preserving accuracy, thus making it a strong candidate for deployment in scalable architectures including cloud infrastructures.