# AlloyBERT: Alloy Property Prediction with Large Language Models

arXiv: https://arxiv.org/abs/2403.19783 · HTML: https://mlsanity.com/paper/2403.19783/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 41 references that are in the map.
- First posted on arXiv 28 Mar 2024.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets](https://mlsanity.com/paper/1906.05474/) (2019)
- [Unifying Molecular and Textual Representations via Multi-task Language Modelling](https://mlsanity.com/paper/2301.12586/) (2023)
- [TransPolymer: a Transformer-based language model for polymer property predictions](https://mlsanity.com/paper/2209.01307/) (2022)
- [Crystal Twins: Self-supervised Learning for Crystalline Material Property Prediction](https://mlsanity.com/paper/2205.01893/) (2022)
- [MOFormer: Self-Supervised Transformer model for Metal-Organic Framework Property Prediction](https://mlsanity.com/paper/2210.14188/) (2022)
- [PeptideBERT: A Language Model based on Transformers for Peptide Property Prediction](https://mlsanity.com/paper/2309.03099/) (2023)
- [Multimodal Language and Graph Learning of Adsorption Configuration in Catalysis](https://mlsanity.com/paper/2401.07408/) (2024)
- [Materials Informatics Transformer: A Language Model for Interpretable Materials Properties Prediction](https://mlsanity.com/paper/2308.16259/) (2023)
- [Attention Visualizer Package: Revealing Word Importance for Deeper Insight into Encoder-Only Transformer Models](https://mlsanity.com/paper/2308.14850/) (2023)

## Built on by (most cited first, 5 of 5)

- [PolyJarvis: An LLM-Orchestrated Agent for Automated All-Atom Molecular Dynamics of Amorphous Homopolymers](https://mlsanity.com/paper/2604.02537/) (2026)
- [SiPhy: Single-Image Physical Property Reasoning](https://mlsanity.com/paper/2607.22355/) (2026)
- [SciQu: Accelerating Materials Properties Prediction with Automated Literature Mining for Self-Driving Laboratories](https://mlsanity.com/paper/2407.08270/) (2024)
- [Text to Band Gap: Pre-trained Language Models as Encoders for Semiconductor Band Gap Prediction](https://mlsanity.com/paper/2501.03456/) (2025)
- [Semantic Embeddings of Chemical Elements for Enhanced Materials Inference and Discovery](https://mlsanity.com/paper/2502.14912/) (2025)

## Abstract (arXiv)

The pursuit of novel alloys tailored to specific requirements poses significant challenges for researchers in the field. This underscores the importance of developing predictive techniques for essential physical properties of alloys based on their chemical composition and processing parameters. This study introduces AlloyBERT, a transformer encoder-based model designed to predict properties such as elastic modulus and yield strength of alloys using textual inputs. Leveraging the pre-trained RoBERTa encoder model as its foundation, AlloyBERT employs self-attention mechanisms to establish meaningful relationships between words, enabling it to interpret human-readable input and predict target alloy properties. By combining a tokenizer trained on our textual data and a RoBERTa encoder pre-trained and fine-tuned for this specific task, we achieved a mean squared error (MSE) of 0.00015 on the Multi Principal Elemental Alloys (MPEA) data set and 0.00611 on the Refractory Alloy Yield Strength (RAYS) dataset. This surpasses the performance of shallow models, which achieved a best-case MSE of 0.00025 and 0.0076 on the MPEA and RAYS datasets respectively. Our results highlight the potential of language models in material science and establish a foundational framework for text-based prediction of alloy properties that does not rely on complex underlying representations, calculations, or simulations.