# MatQnA: A Benchmark Dataset for Multi-modal Large Language Models in Materials Characterization and Analysis

arXiv: https://arxiv.org/abs/2509.11335 · HTML: https://mlsanity.com/paper/2509.11335/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 8 of its 33 references that are in the map.
- First posted on arXiv 14 Sep 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Chart understanding and anomaly detection](https://mlsanity.com/mllm/chart-understanding-and-anomaly-detection/)

## Builds on (most cited first, 8 of 8)

- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark](https://mlsanity.com/paper/2310.18018/) (2023)
- [SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference](https://mlsanity.com/paper/1808.05326/) (2018)
- [Investigating Data Contamination in Modern Benchmarks for Large Language Models](https://mlsanity.com/paper/2311.09783/) (2023)
- [A Survey on Data Contamination for Large Language Models](https://mlsanity.com/paper/2502.14425/) (2025)
- [Benchmarking large language models for materials synthesis: the case of atomic layer deposition](https://mlsanity.com/paper/2412.10477/) (2024)
- [Lessons from the trenches on evaluating machine-learning systems in materials science](https://mlsanity.com/paper/2503.10837/) (2025)

## Built on by (most cited first, 3 of 3)

- [OmniMatBench: A Human-Calibrated Multimodal Reasoning Benchmark Across 19 Materials Science Subfields](https://mlsanity.com/paper/2605.29833/) (2026)
- [MatMMExtract: An Open-Source Pipeline for Panel-Level Extraction of Grounded Image-Text Pairs from Materials Science Literature](https://mlsanity.com/paper/2606.29667/) (2026)
- [QuPAINT: Physics-Aware Multimodal Reasoning for Quantum Material Characterization](https://mlsanity.com/paper/2609.12202/) (2026)

## Abstract (arXiv)

Recently, large language models (LLMs) have achieved remarkable breakthroughs in general domains such as programming and writing, and have demonstrated strong potential in various scientific research scenarios. However, the capabilities of AI models in the highly specialized field of materials characterization and analysis have not yet been systematically or sufficiently validated. To address this gap, we present MatQnA, the first multi-modal benchmark dataset specifically designed for material characterization techniques. MatQnA includes ten mainstream characterization methods, such as X-ray Photoelectron Spectroscopy (XPS), X-ray Diffraction (XRD), Scanning Electron Microscopy (SEM), Transmission Electron Microscopy (TEM), etc. We employ a hybrid approach combining LLMs with human-in-the-loop validation to construct high-quality question-answer pairs, integrating both multiple-choice and subjective questions. Our preliminary evaluation results show that the most advanced multi-modal AI models (e.g., GPT-4.1, Claude 4, Gemini 2.5, and Doubao Vision Pro 32K) have already achieved nearly 90% accuracy on objective questions in materials data interpretation and analysis tasks, demonstrating strong potential for applications in materials characterization and analysis. The MatQnA dataset is publicly available at https://huggingface.co/datasets/richardhzgg/matQnA.