# Mining experimental data from Materials Science literature with Large Language Models: an evaluation study arXiv: https://arxiv.org/abs/2401.11052 ยท HTML: https://mlsanity.com/paper/2401.11052/ ## Key facts - Cited by 11 papers in the map (data through 14 Oct 2026). - Builds on 12 of its 41 references that are in the map. - First posted on arXiv 19 Jan 2024. ## Builds on (most cited first, 12 of 12) - [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019) - [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023) - [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019) - [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022) - [On the Planning Abilities of Large Language Models : A Critical Investigation](https://mlsanity.com/paper/2305.15771/) (2023) - [ChatGPT: Jack of all trades, master of none](https://mlsanity.com/paper/2302.10724/) (2023) - [Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!](https://mlsanity.com/paper/2303.08559/) (2023) - [PEARL: Prompting Large Language Models to Plan and Execute Actions Over Long Documents](https://mlsanity.com/paper/2305.14564/) (2023) - [One Small Step for Generative AI, One Giant Leap for AGI: A Complete Survey on ChatGPT in AIGC Era](https://mlsanity.com/paper/2304.06488/) (2023) - [GPT-3 Models are Poor Few-Shot Learners in the Biomedical Domain](https://mlsanity.com/paper/2109.02555/) (2021) - [MatSciRE: Leveraging Pointer Networks to Automate Entity and Relation Extraction for Material Science Knowledge-base Construction](https://mlsanity.com/paper/2401.09839/) (2024) - [Automatic extraction of materials and properties from superconductors scientific literature](https://mlsanity.com/paper/2210.15600/) (2022) ## Built on by (most cited first, 11 of 11) - [Learning the rules of peptide self-assembly through data mining with large language models](https://mlsanity.com/paper/2411.05421/) (2024) - [Towards an automated workflow in materials science for combining multi-modal simulative and experimental information using data mining and large language models](https://mlsanity.com/paper/2502.14904/) (2025) - [Advances in LLMs with Focus on Reasoning, Adaptability, Efficiency and Ethics](https://mlsanity.com/paper/2506.12365/) (2025) - [AutoDFT: A Closed-Loop Multi-Agent Framework for Autonomous DFT Calculations](https://mlsanity.com/paper/2605.26179/) (2026) - [ComProScanner: A multi-agent based framework for composition-property structured data extraction from scientific literature](https://mlsanity.com/paper/2510.20362/) (2025) - [From Papers to Property Tables: A Priority-Based LLM Workflow for Materials Data Extraction](https://mlsanity.com/paper/2604.07584/) (2026) - [Using large language models to produce literature reviews: Usages and systematic biases of microphysics parametrizations in 2699 publications](https://mlsanity.com/paper/2503.21352/) (2025) - [Accelerating Materials Discovery: Learning a Universal Representation of Chemical Processes for Cross-Domain Property Prediction](https://mlsanity.com/paper/2512.05979/) (2025) - [Intelligent Materials Modelling: Large Language Models Versus Partial Least Squares Regression for Predicting Polysulfone Membrane Mechanical Performance](https://mlsanity.com/paper/2603.13834/) (2026) - [Which Sections of a Research Paper Best Reveal Its Research Methods? Evidence from Library and Information Science](https://mlsanity.com/paper/2606.19051/) (2026) - [Adapt Semantics, Not Structure: Few-Instance Schema Calibration for Scientific PDF Extraction](https://mlsanity.com/paper/2609.34841/) (2026) ## Abstract (arXiv) This study is dedicated to assessing the capabilities of large language models (LLMs) such as GPT-3.5-Turbo, GPT-4, and GPT-4-Turbo in extracting structured information from scientific documents in materials science. To this end, we primarily focus on two critical tasks of information extraction: (i) a named entity recognition (NER) of studied materials and physical properties and (ii) a relation extraction (RE) between these entities. Due to the evident lack of datasets within Materials Informatics (MI), we evaluated using SuperMat, based on superconductor research, and MeasEval, a generic measurement evaluation corpus. The performance of LLMs in executing these tasks is benchmarked against traditional models based on the BERT architecture and rule-based approaches (baseline). We introduce a novel methodology for the comparative analysis of intricate material expressions, emphasising the standardisation of chemical formulas to tackle the complexities inherent in materials science information assessment. For NER, LLMs fail to outperform the baseline with zero-shot prompting and exhibit only limited improvement with few-shot prompting. However, a GPT-3.5-Turbo fine-tuned with the appropriate strategy for RE outperforms all models, including the baseline. Without any fine-tuning, GPT-4 and GPT-4-Turbo display remarkable reasoning and relationship extraction capabilities after being provided with merely a couple of examples, surpassing the baseline. Overall, the results suggest that although LLMs demonstrate relevant reasoning skills in connecting concepts, specialised models are currently a better choice for tasks requiring extracting complex domain-specific entities like materials. These insights provide initial guidance applicable to other materials science sub-domains in future work.