# Benchmarking large language models for materials synthesis: the case of atomic layer deposition

arXiv: https://arxiv.org/abs/2412.10477 · HTML: https://mlsanity.com/paper/2412.10477/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 15 references that are in the map.
- First posted on arXiv 13 Dec 2024.

## Builds on (most cited first, 3 of 3)

- [ChemCrow: Augmenting large-language models with chemistry tools](https://mlsanity.com/paper/2304.05376/) (2023)
- [LLM4Mat-Bench: Benchmarking Large Language Models for Materials Property Prediction](https://mlsanity.com/paper/2411.00177/) (2024)
- [MatSciML: A Broad, Multi-Task Benchmark for Solid-State Materials Modeling](https://mlsanity.com/paper/2309.05934/) (2023)

## Built on by (most cited first, 8 of 8)

- [Language Models Enable Data-Augmented Synthesis Planning for Inorganic Materials](https://mlsanity.com/paper/2506.12557/) (2025)
- [EAIRA: Establishing a Methodology for Evaluating AI Models as Scientific Research Assistants](https://mlsanity.com/paper/2502.20309/) (2025)
- [OpenReview Should be Protected and Leveraged as a Community Asset for Research in the Era of Large Language Models](https://mlsanity.com/paper/2505.21537/) (2025)
- [MatQnA: A Benchmark Dataset for Multi-modal Large Language Models in Materials Characterization and Analysis](https://mlsanity.com/paper/2509.11335/) (2025)
- [Performance of AI agents based on reasoning language models on ALD process optimization tasks](https://mlsanity.com/paper/2601.09980/) (2026)
- [LLM Agents for Knowledge Discovery in Atomic Layer Processing](https://mlsanity.com/paper/2509.26201/) (2025)
- [Publishing FAIR and Machine-actionable Reviews in Materials Science: The Case for Symbolic Knowledge in Neuro-symbolic Artificial Intelligence](https://mlsanity.com/paper/2601.05051/) (2026)
- [Evaluating LLM-based AI agents integrated with materials synthesis tools: the case of atomic layer deposition](https://mlsanity.com/paper/2608.29309/) (2026)

## Abstract (arXiv)

In this work we introduce an open-ended question benchmark, ALDbench, to evaluate the performance of large language models (LLMs) in materials synthesis, and in particular in the field of atomic layer deposition, a thin film growth technique used in energy applications and microelectronics. Our benchmark comprises questions with a level of difficulty ranging from graduate level to domain expert current with the state of the art in the field. Human experts reviewed the questions along the criteria of difficulty and specificity, and the model responses along four different criteria: overall quality, specificity, relevance, and accuracy. We ran this benchmark on an instance of OpenAI's GPT-4o. The responses from the model received a composite quality score of 3.7 on a 1 to 5 scale, consistent with a passing grade. However, 36% of the questions received at least one below average score. An in-depth analysis of the responses identified at least five instances of suspected hallucination. Finally, we observed statistically significant correlations between the difficulty of the question and the quality of the response, the difficulty of the question and the relevance of the response, and the specificity of the question and the accuracy of the response as graded by the human experts. This emphasizes the need to evaluate LLMs across multiple criteria beyond difficulty or accuracy.