mlsanityData to 7 Oct 2026

TransPolymer: a Transformer-based language model for polymer property predictions

arXiv 2209.01307 ↗First posted 3 Sep 2022cs.LG physics.chem-phcited by 43

Not on the map. Why?

Abstract (arXiv)

Accurate and efficient prediction of polymer properties is of great significance in polymer design. Conventionally, expensive and time-consuming experiments or simulations are required to evaluate polymer functions. Recently, Transformer models, equipped with self-attention mechanisms, have exhibited superior performance in natural language processing. However, such methods have not been investigated in polymer sciences. Herein, we report TransPolymer, a Transformer-based language model for polymer property prediction. Our proposed polymer tokenizer with chemical awareness enables learning representations from polymer sequences. Rigorous experiments on ten polymer property prediction benchmarks demonstrate the superior performance of TransPolymer. Moreover, we show that TransPolymer benefits from pretraining on large unlabeled dataset via Masked Language Modeling. Experimental results further manifest the important role of self-attention in modeling polymer sequences. We highlight this model as a promising computational tool for promoting rational polymer design and understanding structure-property relationships from a data science view.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Builds on 28 of its 78 references that are in the map. First posted on arXiv 3 Sep 2022.

Builds on

28 of 78 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  6. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  7. 2020Quantifying Attention Flow in Transformerscited by 759
  8. 2015Convolutional Networks on Graphs for Learning Molecular Fingerprintscited by 751
  9. 2019Analyzing Learned Molecular Representations for Property Predictioncited by 373
  10. 2020ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Predictioncited by 332
  11. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  12. 2018Molecular Transformer - A Model for Uncertainty-Calibrated Chemical Reaction Predictioncited by 225
16 more
  1. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  2. 2018Dissecting Contextual Word Embeddings: Architecture and Representationcited by 146
  3. 2020Self-Attention Attribution: Interpreting Information Interactions Inside Transformercited by 124
  4. 2019SMILES Transformer: Pre-trained Molecular Fingerprint for Low Data Drug Discoverycited by 89
  5. 2021Keeping it Simple: Language Models can learn Complex Molecular Distributionscited by 68
  6. 2017"Found in Translation": Predicting Outcomes of Complex Organic Chemistry Reactions using Neural Sequence-to-Sequence Modelscited by 54
  7. 2020Universal Sentence Representation Learning with Conditional Masked Language Modelcited by 34
  8. 2022Crystal Twins: Self-supervised Learning for Crystalline Material Property Predictioncited by 34
  9. 2018Message-passing neural networks for high-throughput polymer screeningcited by 30
  10. 2020Orbital Graph Convolutional Neural Network for Material Property Predictioncited by 29
  11. 2020Polymer Informatics with Multi-Task Learningcited by 29
  12. 2020Polymer Informatics: Current Status and Critical Next Stepscited by 29
  13. 2022A graph representation of molecular ensembles for polymer property predictioncited by 20
  14. 2020Polymers for Extreme Conditions Designed Using Syntax-Directed Variational Autoencoderscited by 12
  15. 2021Accelerating amorphous polymer electrolyte screening by learning to reduce errors in molecular dynamics simulated propertiescited by 3
  16. 2019SMILES-X: autonomous molecular compounds characterization for small datasets without descriptorscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 4; 2024: 12; 2025: 9; 2026: 14.

43 papers.

Sort:

By area:

  1. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  2. 2022polyBERT: A chemical language model to enable fully machine-driven ultrafast polymer informaticscited by 48
  3. 2022MOFormer: Self-Supervised Transformer model for Metal-Organic Framework Property Predictioncited by 28
  4. 2022Bidirectional Generation of Structure and Properties Through a Single Molecular Foundation Modelcited by 25
  5. 2023Bayesian Optimization of Catalysis With In-Context Learningcited by 20
  6. 2023GPT-MolBERTa: GPT Molecular Features Language Model for molecular property predictioncited by 13
  7. 2024Reinforcement Feature Transformation for Polymer Property Performance Predictioncited by 13
  8. 2024MMPolymer: A Multimodal Multitask Pretraining Framework for Polymer Property Predictioncited by 12
  9. 2024Evaluating the Performance and Robustness of LLMs in Materials Science Q&A and Property Predictionscited by 9
  10. 2023Materials Informatics Transformer: A Language Model for Interpretable Materials Properties Predictioncited by 7
  11. 2024PolyCL: Contrastive Learning for Polymer Representation Learning via Explicit and Implicit Augmentationscited by 7
  12. 2024Inverse Design of Copolymers Including Stoichiometry and Chain Architecturecited by 7
31 more
  1. 2023Transferring a molecular foundation model for polymer property predictionscited by 6
  2. 2024AlloyBERT: Alloy Property Prediction with Large Language Modelscited by 5
  3. 2025Unified Cross-Scale 3D Generation and Understanding via Autoregressive Modelingcited by 5
  4. 2025Multimodal machine learning with large language embedding model for polymer property predictioncited by 5
  5. 2026Artificial Intelligence in Materials Science and Engineering: Current Landscape, Key Challenges, and Future Trajectoriecited by 5
  6. 2024Tokenization for Molecular Foundation Modelscited by 4
  7. 2025High-entropy Advantage in Neural Networks' Generalizabilitycited by 4
  8. 2026Tabular foundation models for in-context prediction of molecular propertiescited by 4
  9. 2024Leveraging Chemistry Foundation Models to Facilitate Structure Focused Retrieval Augmented Generation in Multi-Agent Workflows for Catalyst and Materials Designcited by 3
  10. 2025PolyConf: Unlocking Polymer Conformation Generation through Hierarchical Generative Modelscited by 3
  11. 2026Polymer-Agent: Large Language Model Agent for Polymer Designcited by 3
  12. 2024Large language models, physics-based modeling, experimental measurements: the trinity of data-scarce learning of polymer propertiescited by 2
  13. 2024Molecular topological deep learning for polymer property predictioncited by 2
  14. 2022Predicting CO2 Absorption in Ionic Liquids with Molecular Descriptors and Explainable Graph Neural Networkscited by 1
  15. 2024MolTRES: Improving Chemical Language Representation Learning for Molecular Property Predictioncited by 1
  16. 2025Joint Embedding Predictive Architecture for self-supervised pretraining on polymer molecular graphscited by 1
  17. 2026From Natural Language to Materials Discovery:The Materials Knowledge Navigation Agentcited by 1
  18. 2026Controllable Molecular Generative Foundation Modelscited by 1
  19. 2025Multi-modal cascade feature transfer for polymer property predictioncited by 0
  20. 2025MIPS: a Multimodal Infinite Polymer Sequence Pre-training Framework for Polymer Property Predictioncited by 0
  21. 2025A Conformation-Centric Generative Foundation Model for Linear Polymer Modeling and Designcited by 0
  22. 2025PolyRecommender: A Multimodal Recommendation System for Polymer Discoverycited by 0
  23. 2026Teaching and Evaluating LLMs to Reason About Polymer Design Related Taskscited by 0
  24. 2026Multimodal Machine Learning for Soft High-k Elastomers under Data Scarcitycited by 0
  25. 2026PolyMon: A Unified Framework for Polymer Property Predictioncited by 0
  26. 2026PolyFusionAgent: A Multimodal Foundation Model and Autonomous AI Assistant for Polymer Property Prediction and Inverse Designcited by 0
  27. 2026Periodic Topological Deep Learning for Polymer Design and Discoverycited by 0
  28. 2026UniPolymer: A Unified Framework for Property Prediction, Structure Recommendation, and Evaluation in Polyimide Designcited by 0
  29. 2026PolymerGPT: Multi-property Optimization with a Decoder-Based GPT Model for Generative Polymer Designcited by 0
  30. 2026HiPoly: a hierarchical polymer-native AI framework for property prediction and generative designcited by 0
  31. 2026Molecular representation shapes the balance between target fidelity and exploration in flow based polymer generationcited by 0