# Semantic Embeddings of Chemical Elements for Enhanced Materials Inference and Discovery

arXiv: https://arxiv.org/abs/2502.14912 · HTML: https://mlsanity.com/paper/2502.14912/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 57 references that are in the map.
- First posted on arXiv 19 Feb 2025.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [A Review of Large Language Models and Autonomous Agents in Chemistry](https://mlsanity.com/paper/2407.01603/) (2024)
- [Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineering](https://mlsanity.com/paper/2303.05352/) (2023)
- [ChatGPT Chemistry Assistant for Text Mining and Prediction of MOF Synthesis](https://mlsanity.com/paper/2306.11296/) (2023)
- [polyBERT: A chemical language model to enable fully machine-driven ultrafast polymer informatics](https://mlsanity.com/paper/2209.14803/) (2022)
- [A general-purpose material property data extraction pipeline from large polymer corpora using Natural Language Processing](https://mlsanity.com/paper/2209.13136/) (2022)
- [Understanding LLM Embeddings for Regression](https://mlsanity.com/paper/2411.14708/) (2024)
- [SsciBERT: A Pre-trained Language Model for Social Science Texts](https://mlsanity.com/paper/2206.04510/) (2022)
- [AlloyBERT: Alloy Property Prediction with Large Language Models](https://mlsanity.com/paper/2403.19783/) (2024)

## Abstract (arXiv)

We present a framework for generating universal semantic embeddings of chemical elements to advance materials inference and discovery. This framework leverages ElementBERT, a domain-specific BERT-based natural language processing model trained on 1.29 million abstracts of alloy-related scientific papers, to capture latent knowledge and contextual relationships specific to alloys. These semantic embeddings serve as robust elemental descriptors, consistently outperforming traditional empirical descriptors with significant improvements across multiple downstream tasks. These include predicting mechanical and transformation properties, classifying phase structures, and optimizing materials properties via Bayesian optimization. Applications to titanium alloys, high-entropy alloys, and shape memory alloys demonstrate up to 23% gains in prediction accuracy. Our results show that ElementBERT surpasses general-purpose BERT variants by encoding specialized alloy knowledge. By bridging contextual insights from scientific literature with quantitative inference, our framework accelerates the discovery and optimization of advanced materials, with potential applications extending beyond alloys to other material classes.