# MMPolymer: A Multimodal Multitask Pretraining Framework for Polymer Property Prediction

arXiv: https://arxiv.org/abs/2406.04727 · HTML: https://mlsanity.com/paper/2406.04727/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 61 references that are in the map.
- First posted on arXiv 7 Jun 2024.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [Self-supervised Learning: Generative or Contrastive](https://mlsanity.com/paper/2006.08218/) (2020)
- [SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks](https://mlsanity.com/paper/2006.10503/) (2020)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [ChemRL-GEM: Geometry Enhanced Molecular Representation Learning for Property Prediction](https://mlsanity.com/paper/2106.06130/) (2021)
- [ChemBERTa-2: Towards Chemical Foundation Models](https://mlsanity.com/paper/2209.01712/) (2022)
- [polyBERT: A chemical language model to enable fully machine-driven ultrafast polymer informatics](https://mlsanity.com/paper/2209.14803/) (2022)
- [Unified 2D and 3D Pre-Training of Molecular Representations](https://mlsanity.com/paper/2207.08806/) (2022)
- [TransPolymer: a Transformer-based language model for polymer property predictions](https://mlsanity.com/paper/2209.01307/) (2022)
- [Polymer Informatics with Multi-Task Learning](https://mlsanity.com/paper/2010.15166/) (2020)
- [Polymer Informatics: Current Status and Critical Next Steps](https://mlsanity.com/paper/2011.00508/) (2020)
- [Automated 3D Pre-Training for Molecular Property Prediction](https://mlsanity.com/paper/2306.07812/) (2023)
- [Transferring a molecular foundation model for polymer property predictions](https://mlsanity.com/paper/2310.16958/) (2023)
- [PEMP: Leveraging Physics Properties to Enhance Molecular Property Prediction](https://mlsanity.com/paper/2211.01978/) (2022)

## Built on by (most cited first, 12 of 12)

- [Are High-Degree Representations Really Unnecessary in Equivariant Graph Neural Networks?](https://mlsanity.com/paper/2410.11443/) (2024)
- [Unified Cross-Scale 3D Generation and Understanding via Autoregressive Modeling](https://mlsanity.com/paper/2503.16278/) (2025)
- [Multimodal machine learning with large language embedding model for polymer property prediction](https://mlsanity.com/paper/2503.22962/) (2025)
- [WGFormer: An SE(3)-Transformer Driven by Wasserstein Gradient Flows for Molecular Ground-State Conformation Prediction](https://mlsanity.com/paper/2410.09795/) (2024)
- [PolyConf: Unlocking Polymer Conformation Generation through Hierarchical Generative Models](https://mlsanity.com/paper/2504.08859/) (2025)
- [Universally Invariant Learning in Equivariant GNNs](https://mlsanity.com/paper/2510.13169/) (2025)
- [Geometric Mixture Models for Electrolyte Conductivity Prediction](https://mlsanity.com/paper/2510.15403/) (2025)
- [MIPS: a Multimodal Infinite Polymer Sequence Pre-training Framework for Polymer Property Prediction](https://mlsanity.com/paper/2507.20326/) (2025)
- [A Conformation-Centric Generative Foundation Model for Linear Polymer Modeling and Design](https://mlsanity.com/paper/2510.16023/) (2025)
- [Multimodal Machine Learning for Soft High-k Elastomers under Data Scarcity](https://mlsanity.com/paper/2601.18032/) (2026)
- [PolyFusionAgent: A Multimodal Foundation Model and Autonomous AI Assistant for Polymer Property Prediction and Inverse Design](https://mlsanity.com/paper/2605.26543/) (2026)
- [Periodic Topological Deep Learning for Polymer Design and Discovery](https://mlsanity.com/paper/2605.26833/) (2026)

## Abstract (arXiv)

Polymers are high-molecular-weight compounds constructed by the covalent bonding of numerous identical or similar monomers so that their 3D structures are complex yet exhibit unignorable regularity. Typically, the properties of a polymer, such as plasticity, conductivity, bio-compatibility, and so on, are highly correlated with its 3D structure. However, existing polymer property prediction methods heavily rely on the information learned from polymer SMILES sequences (P-SMILES strings) while ignoring crucial 3D structural information, resulting in sub-optimal performance. In this work, we propose MMPolymer, a novel multimodal multitask pretraining framework incorporating polymer 1D sequential and 3D structural information to encourage downstream polymer property prediction tasks. Besides, considering the scarcity of polymer 3D data, we further introduce the "Star Substitution" strategy to extract 3D structural information effectively. During pretraining, in addition to predicting masked tokens and recovering clear 3D coordinates, MMPolymer achieves the cross-modal alignment of latent representations. Then we further fine-tune the pretrained MMPolymer for downstream polymer property prediction tasks in the supervised learning paradigm. Experiments show that MMPolymer achieves state-of-the-art performance in downstream property prediction tasks. Moreover, given the pretrained MMPolymer, utilizing merely a single modality in the fine-tuning phase can also outperform existing methods, showcasing the exceptional capability of MMPolymer in polymer feature extraction and utilization.