# Large language models, physics-based modeling, experimental measurements: the trinity of data-scarce learning of polymer properties

arXiv: https://arxiv.org/abs/2407.02770 · HTML: https://mlsanity.com/paper/2407.02770/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 57 references that are in the map.
- First posted on arXiv 3 Jul 2024.

## Builds on (most cited first, 16 of 16)

- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [Large-Scale Chemical Language Representations Capture Molecular Structure and Properties](https://mlsanity.com/paper/2106.09553/) (2021)
- [DeepDTA: Deep Drug-Target Binding Affinity Prediction](https://mlsanity.com/paper/1801.10193/) (2018)
- [polyBERT: A chemical language model to enable fully machine-driven ultrafast polymer informatics](https://mlsanity.com/paper/2209.14803/) (2022)
- [Molformer: Motif-based Transformer on 3D Heterogeneous Molecular Graphs](https://mlsanity.com/paper/2110.01191/) (2021)
- [TransPolymer: a Transformer-based language model for polymer property predictions](https://mlsanity.com/paper/2209.01307/) (2022)
- [SMILES2Vec: An Interpretable General-Purpose Deep Neural Network for Predicting Chemical Properties](https://mlsanity.com/paper/1712.02034/) (2017)
- [INO: Invariant Neural Operators for Learning Complex Physical Systems with Momentum Conservation](https://mlsanity.com/paper/2212.14365/) (2022)
- [Uncertainty quantification for noisy inputs-outputs in physics-informed neural networks and neural operators](https://mlsanity.com/paper/2311.11262/) (2023)
- [Conformalized-DeepONet: A Distribution-Free Framework for Uncertainty Quantification in Deep Operator Networks](https://mlsanity.com/paper/2402.15406/) (2024)
- [Self-Attention Based Molecule Representation for Predicting Drug-Target Interaction](https://mlsanity.com/paper/1908.06760/) (2019)
- [Relative Molecule Self-Attention Transformer](https://mlsanity.com/paper/2110.05841/) (2021)
- [GP-MoLFormer: A Foundation Model For Molecular Generation](https://mlsanity.com/paper/2405.04912/) (2024)
- [CheMixNet: Mixed DNN Architectures for Predicting Chemical Properties using Multiple Molecular Representations](https://mlsanity.com/paper/1811.08283/) (2018)
- [Discovery of structure-property relations for molecules via hypothesis-driven active learning over the chemical space](https://mlsanity.com/paper/2301.02665/) (2023)
- [Cloud-Based Real-Time Molecular Screening Platform with MolFormer](https://mlsanity.com/paper/2208.06665/) (2022)

## Built on by (most cited first, 2 of 2)

- [A Survey on Data Synthesis and Augmentation for Large Language Models](https://mlsanity.com/paper/2410.12896/) (2024)
- [Large Language Models for Manufacturing](https://mlsanity.com/paper/2410.21418/) (2024)

## Abstract (arXiv)

Large language models (LLMs) bear promise as a fast and accurate material modeling paradigm for evaluation, analysis, and design. Their vast number of trainable parameters necessitates a wealth of data to achieve accuracy and mitigate overfitting. However, experimental measurements are often limited and costly to obtain in sufficient quantities for finetuning. To this end, we present a physics-based training pipeline that tackles the pathology of data scarcity. The core enabler is a physics-based modeling framework that generates a multitude of synthetic data to align the LLM to a physically consistent initial state before finetuning. Our framework features a two-phase training strategy: (1) utilizing the large-in-amount while less accurate synthetic data for supervised pretraining, and (2) finetuning the phase-1 model with limited experimental data. We empirically demonstrate that supervised pretraining is vital to obtaining accurate finetuned LLMs, via the lens of learning polymer flammability metrics where cone calorimeter data is sparse.