# Materials Informatics Transformer: A Language Model for Interpretable Materials Properties Prediction

arXiv: https://arxiv.org/abs/2308.16259 · HTML: https://mlsanity.com/paper/2308.16259/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 68 references that are in the map.
- First posted on arXiv 30 Aug 2023.

## Builds on (most cited first, 22 of 22)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction](https://mlsanity.com/paper/2010.09885/) (2020)
- [Crystal Diffusion Variational Autoencoder for Periodic Material Generation](https://mlsanity.com/paper/2110.06197/) (2021)
- [The Open Catalyst 2020 (OC20) Dataset and Community Challenges](https://mlsanity.com/paper/2010.09990/) (2020)
- [Graph neural networks for materials science and chemistry](https://mlsanity.com/paper/2208.09481/) (2022)
- [ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing](https://mlsanity.com/paper/2007.06225/) (2020)
- [ProGen2: Exploring the Boundaries of Protein Language Models](https://mlsanity.com/paper/2206.13517/) (2022)
- [Periodic Graph Transformers for Crystal Material Property Prediction](https://mlsanity.com/paper/2209.11807/) (2022)
- [Predicting materials properties without crystal structure: Deep representation learning from stoichiometry](https://mlsanity.com/paper/1910.00617/) (2019)
- [TransPolymer: a Transformer-based language model for polymer property predictions](https://mlsanity.com/paper/2209.01307/) (2022)
- [SELFormer: Molecular Representation Learning via SELFIES Language Models](https://mlsanity.com/paper/2304.04662/) (2023)
- [Improving Molecular Contrastive Learning via Faulty Negative Mitigation and Decomposed Fragment Contrast](https://mlsanity.com/paper/2202.09346/) (2022)
- [Efficient Approximations of Complete Interatomic Potentials for Crystal Property Prediction](https://mlsanity.com/paper/2306.10045/) (2023)
- [Crystal Twins: Self-supervised Learning for Crystalline Material Property Prediction](https://mlsanity.com/paper/2205.01893/) (2022)
- [Orbital Graph Convolutional Neural Network for Material Property Prediction](https://mlsanity.com/paper/2008.06415/) (2020)
- [MOFormer: Self-Supervised Transformer model for Metal-Organic Framework Property Prediction](https://mlsanity.com/paper/2210.14188/) (2022)
- [Deep Generative Model for Periodic Graphs](https://mlsanity.com/paper/2201.11932/) (2022)
- [Connectivity Optimized Nested Graph Networks for Crystal Structures](https://mlsanity.com/paper/2302.14102/) (2023)
- [AugLiChem: Data Augmentation Library of Chemical Structures for Machine Learning](https://mlsanity.com/paper/2111.15112/) (2021)
- [Formula graph self-attention network for representation-domain independent materials discovery](https://mlsanity.com/paper/2201.05649/) (2022)

## Built on by (most cited first, 7 of 7)

- [GPT-MolBERTa: GPT Molecular Features Language Model for molecular property prediction](https://mlsanity.com/paper/2310.03030/) (2023)
- [TopoMAS: Large Language Model Driven Topological Materials Multiagent System](https://mlsanity.com/paper/2507.04053/) (2025)
- [AlloyBERT: Alloy Property Prediction with Large Language Models](https://mlsanity.com/paper/2403.19783/) (2024)
- [CLOUD: A Scalable and Physics-Informed Foundation Model for Crystal Representation Learning](https://mlsanity.com/paper/2506.17345/) (2025)
- [Towards Agentic Intelligence for Materials Science](https://mlsanity.com/paper/2602.00169/) (2026)
- [Enhancing Large Language Models with Domain-Specific Knowledge: The Case in Topological Materials](https://mlsanity.com/paper/2409.13732/) (2024)
- [Optimizing Expert-Designed Crystal Graph Networks for Band-Gap Prediction with an Autonomous LLM Research Loop](https://mlsanity.com/paper/2606.29717/) (2026)

## Abstract (arXiv)

Recently, the remarkable capabilities of large language models (LLMs) have been illustrated across a variety of research domains such as natural language processing, computer vision, and molecular modeling. We extend this paradigm by utilizing LLMs for material property prediction by introducing our model Materials Informatics Transformer (MatInFormer). Specifically, we introduce a novel approach that involves learning the grammar of crystallography through the tokenization of pertinent space group information. We further illustrate the adaptability of MatInFormer by incorporating task-specific data pertaining to Metal-Organic Frameworks (MOFs). Through attention visualization, we uncover the key features that the model prioritizes during property prediction. The effectiveness of our proposed model is empirically validated across 14 distinct datasets, hereby underscoring its potential for high throughput screening through accurate material property prediction.