# Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineering

arXiv: https://arxiv.org/abs/2303.05352 · HTML: https://mlsanity.com/paper/2303.05352/

## Key facts

- Cited by 86 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 51 references that are in the map.
- First posted on arXiv 7 Mar 2023.

## Builds on (most cited first, 10 of 10)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Structured information extraction from complex scientific text with fine-tuned large language models](https://mlsanity.com/paper/2212.05238/) (2022)
- [Inorganic Materials Synthesis Planning with Literature-Trained Neural Networks](https://mlsanity.com/paper/1901.00032/) (2018)
- [Flexible, Model-Agnostic Method for Materials Data Extraction from Text Using General Purpose Language Models](https://mlsanity.com/paper/2302.04914/) (2023)

## Built on by (most cited first, 30 of 86)

- [Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models](https://mlsanity.com/paper/2304.01852/) (2023)
- [ChemLLM: A Chemical Large Language Model](https://mlsanity.com/paper/2402.06852/) (2024)
- [JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models](https://mlsanity.com/paper/2501.10868/) (2025)
- [LLaMP: Large Language Model Made Powerful for High-fidelity Materials Knowledge Retrieval and Distillation](https://mlsanity.com/paper/2401.17244/) (2024)
- [Probing the limitations of multimodal language models for chemistry and materials research](https://mlsanity.com/paper/2411.16955/) (2024)
- [CURIE: Evaluating LLMs On Multitask Scientific Long Context Understanding and Reasoning](https://mlsanity.com/paper/2503.13517/) (2025)
- [Is ChatGPT Transforming Academics' Writing Style?](https://mlsanity.com/paper/2404.08627/) (2024)
- [Foundational Large Language Models for Materials Research](https://mlsanity.com/paper/2412.09560/) (2024)
- [Rethinking Data Protection in the (Generative) Artificial Intelligence Era](https://mlsanity.com/paper/2507.03034/) (2025)
- [HyperFree: A Channel-adaptive and Tuning-free Foundation Model for Hyperspectral Remote Sensing Imagery](https://mlsanity.com/paper/2503.21841/) (2025)
- [Agent-based Learning of Materials Datasets from Scientific Literature](https://mlsanity.com/paper/2312.11690/) (2023)
- [A Survey on Human-Centric LLMs](https://mlsanity.com/paper/2411.14491/) (2024)
- [LaPA: Latent Prompt Assist Model For Medical Visual Question Answering](https://mlsanity.com/paper/2404.13039/) (2024)
- [MatTools: Benchmarking Large Language Models for Materials Science Tools](https://mlsanity.com/paper/2505.10852/) (2025)
- [Unlocking the Potential of ChatGPT: A Comprehensive Exploration of its Applications, Advantages, Limitations, and Future Directions in Natural Language Processing](https://mlsanity.com/paper/2304.02017/) (2023)
- [Regression with Large Language Models for Materials and Molecular Property Prediction](https://mlsanity.com/paper/2409.06080/) (2024)
- [MatViX: Multimodal Information Extraction from Visually Rich Articles](https://mlsanity.com/paper/2410.20494/) (2024)
- [Exploring the Potential of Large Language Models for Improving Digital Forensic Investigation Efficiency](https://mlsanity.com/paper/2402.19366/) (2024)
- [Leveraging Vision Capabilities of Multimodal LLMs for Automated Data Extraction from Plots](https://mlsanity.com/paper/2503.12326/) (2025)
- [GPT-4 as Evaluator: Evaluating Large Language Models on Pest Management in Agriculture](https://mlsanity.com/paper/2403.11858/) (2024)
- [The Northeast Materials Database for Magnetic Materials](https://mlsanity.com/paper/2409.15675/) (2024)
- [A General Approach for Determining Applicability Domain of Machine Learning Models](https://mlsanity.com/paper/2406.05143/) (2024)
- [Toward Reliable Ad-hoc Scientific Information Extraction: A Case Study on Two Materials Datasets](https://mlsanity.com/paper/2406.05348/) (2024)
- [Evaluating Search Engines and Large Language Models for Answering Health Questions](https://mlsanity.com/paper/2407.12468/) (2024)
- [deCIFer: Crystal Structure Prediction from Powder Diffraction Data using Autoregressive Language Models](https://mlsanity.com/paper/2502.02189/) (2025)
- [Artificial Intelligence in Materials Science and Engineering: Current Landscape, Key Challenges, and Future Trajectorie](https://mlsanity.com/paper/2601.12554/) (2026)
- [Automating construction contract review using knowledge graph-enhanced large language models](https://mlsanity.com/paper/2309.12132/) (2023)
- [Cost-Efficient Prompt Engineering for Unsupervised Entity Resolution](https://mlsanity.com/paper/2310.06174/) (2023)
- [Large Language Model for Qualitative Research -- A Systematic Mapping Study](https://mlsanity.com/paper/2411.14473/) (2024)
- [Exploring LLMs for Scientific Information Extraction Using The SciEx Framework](https://mlsanity.com/paper/2512.10004/) (2025)

## Abstract (arXiv)

There has been a growing effort to replace manual extraction of data from research papers with automated data extraction based on natural language processing, language models, and recently, large language models (LLMs). Although these methods enable efficient extraction of data from large sets of research papers, they require a significant amount of up-front effort, expertise, and coding. In this work we propose the ChatExtract method that can fully automate very accurate data extraction with minimal initial effort and background, using an advanced conversational LLM. ChatExtract consists of a set of engineered prompts applied to a conversational LLM that both identify sentences with data, extract that data, and assure the data's correctness through a series of follow-up questions. These follow-up questions largely overcome known issues with LLMs providing factually inaccurate responses. ChatExtract can be applied with any conversational LLMs and yields very high quality data extraction. In tests on materials data we find precision and recall both close to 90% from the best conversational LLMs, like ChatGPT-4. We demonstrate that the exceptional performance is enabled by the information retention in a conversational model combined with purposeful redundancy and introducing uncertainty through follow-up prompts. These results suggest that approaches similar to ChatExtract, due to their simplicity, transferability, and accuracy are likely to become powerful tools for data extraction in the near future. Finally, databases for critical cooling rates of metallic glasses and yield strengths of high entropy alloys are developed using ChatExtract.