# Can Multimodal LLMs See Materials Clearly? A Multimodal Benchmark on Materials Characterization

arXiv: https://arxiv.org/abs/2509.09307 · HTML: https://mlsanity.com/paper/2509.09307/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 22 of its 54 references that are in the map.
- First posted on arXiv 11 Sep 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Chart understanding and anomaly detection](https://mlsanity.com/mllm/chart-understanding-and-anomaly-detection/)

## Builds on (most cited first, 22 of 22)

- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling](https://mlsanity.com/paper/2501.17811/) (2025)
- [MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark](https://mlsanity.com/paper/2409.02813/) (2024)
- [PathVQA: 30000+ Questions for Medical Visual Question Answering](https://mlsanity.com/paper/2003.10286/) (2020)
- [PaperQA: Retrieval-Augmented Generative Agent for Scientific Research](https://mlsanity.com/paper/2312.07559/) (2023)
- [SciCap: Generating Captions for Scientific Figures](https://mlsanity.com/paper/2110.11624/) (2021)
- [HoneyComb: A Flexible LLM-Based Agent System for Materials Science](https://mlsanity.com/paper/2409.00135/) (2024)
- [SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretation](https://mlsanity.com/paper/2405.08807/) (2024)
- [Visual Large Language Models for Generalized and Specialized Applications](https://mlsanity.com/paper/2501.02765/) (2025)
- [LLM-Prop: Predicting Physical And Electronic Properties Of Crystalline Solids From Their Text Descriptions](https://mlsanity.com/paper/2310.14029/) (2023)
- [Overview: Computer vision and machine learning for microstructural characterization and analysis](https://mlsanity.com/paper/2005.14260/) (2020)
- [High throughput quantitative metallography for complex microstructures using deep learning: A case study in ultrahigh carbon steel](https://mlsanity.com/paper/1805.08693/) (2018)
- [MatExpert: Decomposing Materials Discovery by Mimicking Human Experts](https://mlsanity.com/paper/2410.21317/) (2024)
- [MatterChat: A Multi-Modal LLM for Material Science](https://mlsanity.com/paper/2502.13107/) (2025)
- [Multi defect detection and analysis of electron microscopy images with deep learning](https://mlsanity.com/paper/2108.08883/) (2021)
- [EXSCLAIM! -- An automated pipeline for the construction of labeled materials imaging datasets from literature](https://mlsanity.com/paper/2103.10631/) (2021)
- [Beyond Human Vision: The Role of Large Vision Language Models in Microscope Image Analysis](https://mlsanity.com/paper/2405.00876/) (2024)

## Built on by (most cited first, 9 of 9)

- [TextAlign: Preference Alignment for Text Rendering with Hierarchical Rewards](https://mlsanity.com/paper/2605.19320/) (2026)
- [LabGuard: Grounding Natural-Language Laboratory Rules into Runtime Guards for Embodied Laboratory Agents](https://mlsanity.com/paper/2606.31045/) (2026)
- [PolyReal: A Benchmark for Real-World Polymer Science Workflows](https://mlsanity.com/paper/2604.02934/) (2026)
- [Text to Automata Diagrams: Comparing TikZ Code Generation with Direct Image Synthesis](https://mlsanity.com/paper/2603.07936/) (2026)
- [MatMMExtract: An Open-Source Pipeline for Panel-Level Extraction of Grounded Image-Text Pairs from Materials Science Literature](https://mlsanity.com/paper/2606.29667/) (2026)
- [SciMIF: Understanding Multimodal Instruction Following in Scientific Domains](https://mlsanity.com/paper/2608.25973/) (2026)
- [Physics as the label for measuring and correcting materials reasoning in multimodal models](https://mlsanity.com/paper/2609.12181/) (2026)
- [QuPAINT: Physics-Aware Multimodal Reasoning for Quantum Material Characterization](https://mlsanity.com/paper/2609.12202/) (2026)
- [LabOPBench: A Multimodal Benchmark for Evidence-Grounded Laboratory Decision Making](https://mlsanity.com/paper/2610.19780/) (2026)

## Abstract (arXiv)

Materials characterization is fundamental to acquiring materials information, revealing the processing-microstructure-property relationships that guide material design and optimization. While multimodal large language models (MLLMs) have recently shown promise in generative and predictive tasks within materials science, their capacity to understand real-world characterization imaging data remains underexplored. To bridge this gap, we present MatCha, the first benchmark for materials characterization image understanding, comprising 1,500 questions that demand expert-level domain expertise. MatCha encompasses four key stages of materials research comprising 21 distinct tasks, each designed to reflect authentic challenges faced by materials scientists. Our evaluation of state-of-the-art MLLMs on MatCha reveals a significant performance gap compared to human experts. These models exhibit degradation when addressing questions requiring higher-level expertise and sophisticated visual perception. Simple few-shot and chain-of-thought prompting struggle to alleviate these limitations. These findings highlight that existing MLLMs still exhibit limited adaptability to real-world materials characterization scenarios. We hope MatCha will facilitate future research in areas such as new material discovery and autonomous scientific agents. MatCha is available at https://github.com/FreedomIntelligence/MatCha.