# Towards an automated workflow in materials science for combining multi-modal simulative and experimental information using data mining and large language models

arXiv: https://arxiv.org/abs/2502.14904 · HTML: https://mlsanity.com/paper/2502.14904/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 53 references that are in the map.
- First posted on arXiv 18 Feb 2025.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Is ChatGPT a Good NLG Evaluator? A Preliminary Study](https://mlsanity.com/paper/2303.04048/) (2023)
- [Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineering](https://mlsanity.com/paper/2303.05352/) (2023)
- [14 Examples of How LLMs Can Transform Materials Science and Chemistry: A Reflection on a Large Language Model Hackathon](https://mlsanity.com/paper/2306.06283/) (2023)
- [Searching for Best Practices in Retrieval-Augmented Generation](https://mlsanity.com/paper/2407.01219/) (2024)
- [ProtAgents: Protein discovery via large language model multi-agent collaborations combining physics and machine learning](https://mlsanity.com/paper/2402.04268/) (2024)
- [Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions](https://mlsanity.com/paper/2305.10435/) (2023)
- [From Concept to Manufacturing: Evaluating Vision-Language Models for Engineering Design](https://mlsanity.com/paper/2311.12668/) (2023)
- [MechGPT, a language-based strategy for mechanics and materials modeling that connects knowledge across scales, disciplines and modalities](https://mlsanity.com/paper/2310.10445/) (2023)
- [Physics Guided Deep Learning for Generative Design of Crystal Materials with Symmetry Constraints](https://mlsanity.com/paper/2203.14352/) (2022)
- [Opportunities for Retrieval and Tool Augmented Large Language Models in Scientific Facilities](https://mlsanity.com/paper/2312.01291/) (2023)
- [Accelerating Scientific Discovery with Generative Knowledge Extraction, Graph-Based Representation, and Multimodal Intelligent Graph Reasoning](https://mlsanity.com/paper/2403.11996/) (2024)
- [Materials science in the era of large language models: a perspective](https://mlsanity.com/paper/2403.06949/) (2024)
- [AMGPT: a Large Language Model for Contextual Querying in Additive Manufacturing](https://mlsanity.com/paper/2406.00031/) (2024)
- [Accelerating Material Design with the Generative Toolkit for Scientific Discovery](https://mlsanity.com/paper/2207.03928/) (2022)
- [Mining experimental data from Materials Science literature with Large Language Models: an evaluation study](https://mlsanity.com/paper/2401.11052/) (2024)
- [Cephalo: Multi-Modal Vision-Language Models for Bio-Inspired Materials Analysis and Design](https://mlsanity.com/paper/2405.19076/) (2024)

## Built on by (most cited first, 3 of 3)

- [Ontology-aligned structuring and reuse of multimodal materials data and workflows towards automatic reproduction](https://mlsanity.com/paper/2601.12582/) (2026)
- [Zero-shot reasoning for simulating scholarly peer-review](https://mlsanity.com/paper/2510.02027/) (2025)
- [Agentic Resource Allocation for Batch Multi-Objective Bayesian Optimization in Autonomous Materials Discovery](https://mlsanity.com/paper/2610.04134/) (2026)

## Abstract (arXiv)

To retrieve and compare scientific data of simulations and experiments in materials science, data needs to be easily accessible and machine readable to qualify and quantify various materials science phenomena. The recent progress in open science leverages the accessibility to data. However, a majority of information is encoded within scientific documents limiting the capability of finding suitable literature as well as material properties. This manuscript showcases an automated workflow, which unravels the encoded information from scientific literature to a machine readable data structure of texts, figures, tables, equations and meta-data, using natural language processing and language as well as vision transformer models to generate a machine-readable database. The machine-readable database can be enriched with local data, as e.g. unpublished or private material data, leading to knowledge synthesis. The study shows that such an automated workflow accelerates information retrieval, proximate context detection and material property extraction from multi-modal input data exemplarily shown for the research field of microstructural analyses of face-centered cubic single crystals. Ultimately, a Retrieval-Augmented Generation (RAG) based Large Language Model (LLM) enables a fast and efficient question answering chat bot.