# Text to Band Gap: Pre-trained Language Models as Encoders for Semiconductor Band Gap Prediction

arXiv: https://arxiv.org/abs/2501.03456 · HTML: https://mlsanity.com/paper/2501.03456/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 39 references that are in the map.
- First posted on arXiv 7 Jan 2025.

## Builds on (most cited first, 9 of 9)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [MatSciBERT: A Materials Domain Language Model for Text Mining and Information Extraction](https://mlsanity.com/paper/2109.15290/) (2021)
- [AMGPT: a Large Language Model for Contextual Querying in Additive Manufacturing](https://mlsanity.com/paper/2406.00031/) (2024)
- [PeptideBERT: A Language Model based on Transformers for Peptide Property Prediction](https://mlsanity.com/paper/2309.03099/) (2023)
- [Multimodal Language and Graph Learning of Adsorption Configuration in Catalysis](https://mlsanity.com/paper/2401.07408/) (2024)
- [Regression with Large Language Models for Materials and Molecular Property Prediction](https://mlsanity.com/paper/2409.06080/) (2024)
- [AlloyBERT: Alloy Property Prediction with Large Language Models](https://mlsanity.com/paper/2403.19783/) (2024)
- [A RESTful API for exchanging Materials Data in the AFLOWLIB.org consortium](https://mlsanity.com/paper/1403.2642/) (2014)

## Abstract (arXiv)

We investigate transformer-based language models, including RoBERTa, T5, Llama-3, and MatSciBERT, for predicting the band gaps of semiconductor materials directly from textual descriptions. The inputs encode key material features, such as chemical composition, crystal system, space group, and other structural and electronic properties. Unlike shallow machine learning models, which require extensive feature engineering, or Graph Neural Networks, which rely on graph representations derived from atomic coordinates, pretrained language models can process textual inputs directly, eliminating the need for manual feature preprocessing or structure-based encoding. Material descriptions were constructed in two formats: structured strings with a consistent template and natural language narratives generated via the ChatGPT API. Each model was augmented with a custom regression head and finetuned for band gap prediction task. Language models of different architectures and parameter sizes were all able to predict band gaps from human-readable text with strong accuracy, achieving MAEs in the range of 0.25-0.33 eV, highlighting the success of this approach for scientific regression tasks. Finetuned Llama-3, with 1.2 billion parameters, achieved the highest accuracy (MAE 0.248 eV, R2 0.891). MatSciBERT, pretrained on materials science literature, reached comparable performance (MAE 0.288 eV, R2 0.871) with significantly fewer parameters (110 million), emphasizing the importance of domain-specific pretraining. Attention analysis shows that both models selectively focus on compositional and spin-related features while de-emphasizing geometric features, reflecting the difficulty of capturing spatial information from text. These results establish that pretrained language models can effectively extract complex feature-property relationships from textual material descriptions.