# MatSci-NLP: Evaluating Scientific Language Models on Materials Science Language Tasks Using Text-to-Schema Modeling

arXiv: https://arxiv.org/abs/2305.08264 · HTML: https://mlsanity.com/paper/2305.08264/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 57 references that are in the map.
- First posted on arXiv 14 May 2023.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers](https://mlsanity.com/paper/2105.03011/) (2021)
- [Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets](https://mlsanity.com/paper/1906.05474/) (2019)
- [Text2Event: Controllable Sequence-to-Structure Generation for End-to-end Event Extraction](https://mlsanity.com/paper/2106.09232/) (2021)
- [SciFive: a text-to-text transformer model for biomedical literature](https://mlsanity.com/paper/2106.03598/) (2021)
- [Multi-task Sequence to Sequence Learning](https://mlsanity.com/paper/1511.06114/) (2015)
- [MatSciBERT: A Materials Domain Language Model for Text Mining and Information Extraction](https://mlsanity.com/paper/2109.15290/) (2021)
- [Enriching Pre-trained Language Model with Entity Information for Relation Classification](https://mlsanity.com/paper/1905.08284/) (2019)
- [BioMegatron: Larger Biomedical Domain Language Model](https://mlsanity.com/paper/2010.06060/) (2020)
- [BERT with History Answer Embedding for Conversational Question Answering](https://mlsanity.com/paper/1905.05412/) (2019)
- [The Materials Science Procedural Text Corpus: Annotating Materials Synthesis Procedures with Shallow Semantic Structures](https://mlsanity.com/paper/1905.06939/) (2019)
- [The SOFC-Exp Corpus and Neural Approaches to Information Extraction in the Materials Science Domain](https://mlsanity.com/paper/2006.03039/) (2020)
- [Inorganic Materials Synthesis Planning with Literature-Trained Neural Networks](https://mlsanity.com/paper/1901.00032/) (2018)
- [Looking Through Glass: Knowledge Discovery from Materials Science Literature using Natural Language Processing](https://mlsanity.com/paper/2101.01508/) (2021)
- [Annotating and Extracting Synthesis Process of All-Solid-State Batteries from Scientific Literature](https://mlsanity.com/paper/2002.07339/) (2020)
- [Pre-training technique to localize medical BERT and enhance biomedical BERT](https://mlsanity.com/paper/2005.07202/) (2020)

## Built on by (most cited first, 30 of 35)

- [A Review of Large Language Models and Autonomous Agents in Chemistry](https://mlsanity.com/paper/2407.01603/) (2024)
- [Agentic AI for Scientific Discovery: A Survey of Progress, Challenges, and Future Directions](https://mlsanity.com/paper/2503.08979/) (2025)
- [HoneyComb: A Flexible LLM-Based Agent System for Materials Science](https://mlsanity.com/paper/2409.00135/) (2024)
- [Are LLMs Ready for Real-World Materials Discovery?](https://mlsanity.com/paper/2402.05200/) (2024)
- [A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers](https://mlsanity.com/paper/2508.21148/) (2025)
- [Are large language models superhuman chemists?](https://mlsanity.com/paper/2404.01475/) (2024)
- [LLaMP: Large Language Model Made Powerful for High-fidelity Materials Knowledge Retrieval and Distillation](https://mlsanity.com/paper/2401.17244/) (2024)
- [LLM-Prop: Predicting Physical And Electronic Properties Of Crystalline Solids From Their Text Descriptions](https://mlsanity.com/paper/2310.14029/) (2023)
- [HoneyBee: Progressive Instruction Finetuning of Large Language Models for Materials Science](https://mlsanity.com/paper/2310.08511/) (2023)
- [MatExpert: Decomposing Materials Discovery by Mimicking Human Experts](https://mlsanity.com/paper/2410.21317/) (2024)
- [Foundational Large Language Models for Materials Research](https://mlsanity.com/paper/2412.09560/) (2024)
- [MatSciML: A Broad, Multi-Task Benchmark for Solid-State Materials Modeling](https://mlsanity.com/paper/2309.05934/) (2023)
- [Reconstructing Materials Tetrahedron: Challenges in Materials Information Extraction](https://mlsanity.com/paper/2310.08383/) (2023)
- [A Survey of AI for Materials Science: Foundation Models, LLM Agents, Datasets, and Tools](https://mlsanity.com/paper/2506.20743/) (2025)
- [MELT: Materials-aware Continued Pre-training for Language Model Adaptation to Materials Science](https://mlsanity.com/paper/2410.15126/) (2024)
- [TritonDFT: Automating DFT with a Multi-Agent Framework](https://mlsanity.com/paper/2603.03372/) (2026)
- [FoodPuzzle: Developing Large Language Model Agents as Flavor Scientists](https://mlsanity.com/paper/2409.12832/) (2024)
- [MSQA: Benchmarking LLMs on Graduate-Level Materials Science Reasoning and Knowledge](https://mlsanity.com/paper/2505.23982/) (2025)
- [Perovskite-LLM: Knowledge-Enhanced Large Language Models for Perovskite Solar Cell Research](https://mlsanity.com/paper/2502.12669/) (2025)
- [Towards Fully-Automated Materials Discovery via Large-Scale Synthesis Dataset and Expert-Level LLM-as-a-Judge](https://mlsanity.com/paper/2502.16457/) (2025)
- [Incorporating Domain Knowledge into Materials Tokenization](https://mlsanity.com/paper/2506.11115/) (2025)
- [CrystalXRD-Bench: Benchmarking Vision-Language Models for XRD Peak Indexing Across Diverse Crystalline Materials](https://mlsanity.com/paper/2605.29446/) (2026)
- [On the importance of catalyst-adsorbate 3D interactions for relaxed energy predictions](https://mlsanity.com/paper/2310.06682/) (2023)
- [Quokka: An Open-source Large Language Model ChatBot for Material Science](https://mlsanity.com/paper/2401.01089/) (2024)
- [Extracting Polymer Nanocomposite Samples from Full-Length Documents](https://mlsanity.com/paper/2403.00260/) (2024)
- [Efficient Scientific Full Text Classification: The Case of EICAT Impact Assessments](https://mlsanity.com/paper/2502.06551/) (2025)
- [ScholarBench: A Bilingual Benchmark for Abstraction, Comprehension, and Reasoning Evaluation in Academic Contexts](https://mlsanity.com/paper/2505.16566/) (2025)
- [STAYKATE: Hybrid In-Context Example Selection Combining Representativeness Sampling and Retrieval-based Approach -- A Case Study on Science Domains](https://mlsanity.com/paper/2412.20043/) (2024)
- [Mechanisms of Matter: Language Inferential Benchmark on Physicochemical Hypothesis in Materials Synthesis](https://mlsanity.com/paper/2509.25281/) (2025)
- [Sustainable Materials Discovery in the Era of Artificial Intelligence](https://mlsanity.com/paper/2601.21527/) (2026)

## Abstract (arXiv)

We present MatSci-NLP, a natural language benchmark for evaluating the performance of natural language processing (NLP) models on materials science text. We construct the benchmark from publicly available materials science text data to encompass seven different NLP tasks, including conventional NLP tasks like named entity recognition and relation classification, as well as NLP tasks specific to materials science, such as synthesis action retrieval which relates to creating synthesis procedures for materials. We study various BERT-based models pretrained on different scientific text corpora on MatSci-NLP to understand the impact of pretraining strategies on understanding materials science text. Given the scarcity of high-quality annotated data in the materials science domain, we perform our fine-tuning experiments with limited training data to encourage the generalize across MatSci-NLP tasks. Our experiments in this low-resource training setting show that language models pretrained on scientific text outperform BERT trained on general text. MatBERT, a model pretrained specifically on materials science journals, generally performs best for most tasks. Moreover, we propose a unified text-to-schema for multitask learning on benchmark and compare its performance with traditional fine-tuning methods. In our analysis of different training methods, we find that our proposed text-to-schema methods inspired by question-answering consistently outperform single and multitask NLP fine-tuning methods. The code and datasets are publicly available at urlhttps://github.com/BangLab-UdeM-Mila/NLP4MatSci-ACL23.