mlsanityData to 14 Oct 2026

MatQnA: A Benchmark Dataset for Multi-modal Large Language Models in Materials Characterization and Analysis

arXiv 2509.11335 ↗First posted 14 Sep 2025cs.LG cond-mat.mtrl-scicited by 3

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

Recently, large language models (LLMs) have achieved remarkable breakthroughs in general domains such as programming and writing, and have demonstrated strong potential in various scientific research scenarios. However, the capabilities of AI models in the highly specialized field of materials characterization and analysis have not yet been systematically or sufficiently validated. To address this gap, we present MatQnA, the first multi-modal benchmark dataset specifically designed for material characterization techniques. MatQnA includes ten mainstream characterization methods, such as X-ray Photoelectron Spectroscopy (XPS), X-ray Diffraction (XRD), Scanning Electron Microscopy (SEM), Transmission Electron Microscopy (TEM), etc. We employ a hybrid approach combining LLMs with human-in-the-loop validation to construct high-quality question-answer pairs, integrating both multiple-choice and subjective questions. Our preliminary evaluation results show that the most advanced multi-modal AI models (e.g., GPT-4.1, Claude 4, Gemini 2.5, and Doubao Vision Pro 32K) have already achieved nearly 90% accuracy on objective questions in materials data interpretation and analysis tasks, demonstrating strong potential for applications in materials characterization and analysis. The MatQnA dataset is publicly available at https://huggingface.co/datasets/richardhzgg/matQnA.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 8 of its 33 references that are in the map. First posted on arXiv 14 Sep 2025.

Builds on

8 of 33 references.

References · 8

  1. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  2. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  3. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 348
  4. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  5. 2023Investigating Data Contamination in Modern Benchmarks for Large Language Modelscited by 138
  6. 2025A Survey on Data Contamination for Large Language Modelscited by 35
  7. 2024Benchmarking large language models for materials synthesis: the case of atomic layer depositioncited by 8
  8. 2025Lessons from the trenches on evaluating machine-learning systems in materials sciencecited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026OmniMatBench: A Human-Calibrated Multimodal Reasoning Benchmark Across 19 Materials Science Subfieldscited by 0
  2. 2026MatMMExtract: An Open-Source Pipeline for Panel-Level Extraction of Grounded Image-Text Pairs from Materials Science Literaturecited by 0
  3. 2026QuPAINT: Physics-Aware Multimodal Reasoning for Quantum Material Characterizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MaterialFigBENCH: benchmark dataset with figures for evaluating college-level materials science problem-solving abilities of multimodal large language models
  2. 2025MatterChat: A Multi-Modal LLM for Material Science
  3. 2025MatSciBench: Benchmarking the Reasoning Ability of Large Language Models in Materials Science
  4. 2025Role of Large Language Models and Retrieval-Augmented Generation for Accelerating Crystalline Material Discovery: A Systematic Review
  5. 2024Evaluating the Performance and Robustness of LLMs in Materials Science Q&A and Property Predictions
  6. 2025Large language models in materials science and the need for open-source approaches
  7. 2024Are LLMs Ready for Real-World Materials Discovery?
  8. 2024Foundational Large Language Models for Materials Research