# MatSciBERT: A Materials Domain Language Model for Text Mining and Information Extraction

arXiv: https://arxiv.org/abs/2109.15290 · HTML: https://mlsanity.com/paper/2109.15290/

## Key facts

- Cited by 83 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 60 references that are in the map.
- First posted on arXiv 30 Sep 2021.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [FinBERT: Financial Sentiment Analysis with Pre-trained Language Models](https://mlsanity.com/paper/1908.10063/) (2019)
- [Matching the Blanks: Distributional Similarity for Relation Learning](https://mlsanity.com/paper/1906.03158/) (2019)
- [Construction of the Literature Graph in Semantic Scholar](https://mlsanity.com/paper/1805.02262/) (2018)
- [Parametric UMAP embeddings for representation and semi-supervised learning](https://mlsanity.com/paper/2009.12981/) (2020)
- [On the Language Neutrality of Pre-trained Multilingual Representations](https://mlsanity.com/paper/2004.05160/) (2020)
- [The Materials Science Procedural Text Corpus: Annotating Materials Synthesis Procedures with Shallow Semantic Structures](https://mlsanity.com/paper/1905.06939/) (2019)
- [The SOFC-Exp Corpus and Neural Approaches to Information Extraction in the Materials Science Domain](https://mlsanity.com/paper/2006.03039/) (2020)
- [Looking Through Glass: Knowledge Discovery from Materials Science Literature using Natural Language Processing](https://mlsanity.com/paper/2101.01508/) (2021)
- [MatScIE: An automated tool for the generation of databases of methods and parameters used in the computational materials science literature](https://mlsanity.com/paper/2009.06819/) (2020)
- [Why and when should you pool? Analyzing Pooling in Recurrent Architectures](https://mlsanity.com/paper/2005.00159/) (2020)

## Built on by (most cited first, 30 of 83)

- [K2: A Foundation Language Model for Geoscience Knowledge Understanding and Utilization](https://mlsanity.com/paper/2306.05064/) (2023)
- [Are LLMs Ready for Real-World Materials Discovery?](https://mlsanity.com/paper/2402.05200/) (2024)
- [MatSci-NLP: Evaluating Scientific Language Models on Materials Science Language Tasks Using Text-to-Schema Modeling](https://mlsanity.com/paper/2305.08264/) (2023)
- [LLaMP: Large Language Model Made Powerful for High-fidelity Materials Knowledge Retrieval and Distillation](https://mlsanity.com/paper/2401.17244/) (2024)
- [Large Language Models as Master Key: Unlocking the Secrets of Materials Science with GPT](https://mlsanity.com/paper/2304.02213/) (2023)
- [LLM-Prop: Predicting Physical And Electronic Properties Of Crystalline Solids From Their Text Descriptions](https://mlsanity.com/paper/2310.14029/) (2023)
- [Materials science in the era of large language models: a perspective](https://mlsanity.com/paper/2403.06949/) (2024)
- [A general-purpose material property data extraction pipeline from large polymer corpora using Natural Language Processing](https://mlsanity.com/paper/2209.13136/) (2022)
- [HoneyBee: Progressive Instruction Finetuning of Large Language Models for Materials Science](https://mlsanity.com/paper/2310.08511/) (2023)
- [ControlAgent: Automating Control System Design via Novel Integration of LLM Agents and Domain Expertise](https://mlsanity.com/paper/2410.19811/) (2024)
- [Foundational Large Language Models for Materials Research](https://mlsanity.com/paper/2412.09560/) (2024)
- [DiSCoMaT: Distantly Supervised Composition Extraction from Tables in Materials Science Articles](https://mlsanity.com/paper/2207.01079/) (2022)
- [An Interdisciplinary Outlook on Large Language Models for Scientific Research](https://mlsanity.com/paper/2311.04929/) (2023)
- [Periodic Materials Generation using Text-Guided Joint Diffusion Model](https://mlsanity.com/paper/2503.00522/) (2025)
- [How Far Are AI Scientists from Changing the World?](https://mlsanity.com/paper/2507.23276/) (2025)
- [GPT-MolBERTa: GPT Molecular Features Language Model for molecular property prediction](https://mlsanity.com/paper/2310.03030/) (2023)
- [Pre-trained Language Models for Keyphrase Generation: A Thorough Empirical Study](https://mlsanity.com/paper/2212.10233/) (2022)
- [LLEMA: Evolutionary Search with LLMs for Multi-Objective Materials Discovery](https://mlsanity.com/paper/2510.22503/) (2025)
- [ChemTEB: Chemical Text Embedding Benchmark, an Overview of Embedding Models Performance & Efficiency on a Specific Domain](https://mlsanity.com/paper/2412.00532/) (2024)
- [MaScQA: A Question Answering Dataset for Investigating Materials Science Knowledge of Large Language Models](https://mlsanity.com/paper/2308.09115/) (2023)
- [MatSciRE: Leveraging Pointer Networks to Automate Entity and Relation Extraction for Material Science Knowledge-base Construction](https://mlsanity.com/paper/2401.09839/) (2024)
- [MatChat: A Large Language Model and Application Service Platform for Materials Science](https://mlsanity.com/paper/2310.07197/) (2023)
- [CheX-GPT: Harnessing Large Language Models for Enhanced Chest X-ray Report Labeling](https://mlsanity.com/paper/2401.11505/) (2024)
- [AI-driven inverse design of materials: Past, present and future](https://mlsanity.com/paper/2411.09429/) (2024)
- [Evaluating the Knowledge Dependency of Questions](https://mlsanity.com/paper/2211.11902/) (2022)
- [Flexible, Model-Agnostic Method for Materials Data Extraction from Text Using General Purpose Language Models](https://mlsanity.com/paper/2302.04914/) (2023)
- [Evaluating the Performance and Robustness of LLMs in Materials Science Q&A and Property Predictions](https://mlsanity.com/paper/2409.14572/) (2024)
- [34 Examples of LLM Applications in Materials Science and Chemistry: Towards Automation, Assistants, Agents, and Accelerated Scientific Discovery](https://mlsanity.com/paper/2505.03049/) (2025)
- [Predicting New Research Directions in Materials Science using Large Language Models and Concept Graphs](https://mlsanity.com/paper/2506.16824/) (2025)
- [Reconstructing Materials Tetrahedron: Challenges in Materials Information Extraction](https://mlsanity.com/paper/2310.08383/) (2023)

## Abstract (arXiv)

An overwhelmingly large amount of knowledge in the materials domain is generated and stored as text published in peer-reviewed scientific literature. Recent developments in natural language processing, such as bidirectional encoder representations from transformers (BERT) models, provide promising tools to extract information from these texts. However, direct application of these models in the materials domain may yield suboptimal results as the models themselves may not be trained on notations and jargon that are specific to the domain. Here, we present a materials-aware language model, namely, MatSciBERT, which is trained on a large corpus of scientific literature published in the materials domain. We further evaluate the performance of MatSciBERT on three downstream tasks, namely, abstract classification, named entity recognition, and relation extraction, on different materials datasets. We show that MatSciBERT outperforms SciBERT, a language model trained on science corpus, on all the tasks. Further, we discuss some of the applications of MatSciBERT in the materials domain for extracting information, which can, in turn, contribute to materials discovery or optimization. Finally, to make the work accessible to the larger materials community, we make the pretrained and finetuned weights and the models of MatSciBERT freely accessible.