# MatPhaseBench: A Semantics-Guided Benchmark for Materials Phase Diagrams Understanding

arXiv: https://arxiv.org/abs/2607.02934 · HTML: https://mlsanity.com/paper/2607.02934/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 37 references that are in the map.
- First posted on arXiv 3 Jul 2026.

## Builds on (most cited first, 23 of 23)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [OpenAI GPT-5 System Card](https://mlsanity.com/paper/2601.03267/) (2025)
- [MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark](https://mlsanity.com/paper/2409.02813/) (2024)
- [MinerU: An Open-Source Solution for Precise Document Content Extraction](https://mlsanity.com/paper/2409.18839/) (2024)
- [FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI](https://mlsanity.com/paper/2411.04872/) (2024)
- [SciCode: A Research Coding Benchmark Curated by Scientists](https://mlsanity.com/paper/2407.13168/) (2024)
- [EvoChart: A Benchmark and a Self-Training Approach Towards Real-World Chart Understanding](https://mlsanity.com/paper/2409.01577/) (2024)
- [MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Research](https://mlsanity.com/paper/2503.13399/) (2025)
- [DocVLM: Make Your VLM an Efficient Reader](https://mlsanity.com/paper/2412.08746/) (2024)
- [MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models](https://mlsanity.com/paper/2504.05782/) (2025)
- [EncQA: Benchmarking Vision-Language Models on Visual Encodings for Charts](https://mlsanity.com/paper/2508.04650/) (2025)
- [MAC: A Live Benchmark for Multimodal Large Language Models in Scientific Understanding](https://mlsanity.com/paper/2508.15802/) (2025)
- [Cephalo: Multi-Modal Vision-Language Models for Bio-Inspired Materials Analysis and Design](https://mlsanity.com/paper/2405.19076/) (2024)
- [MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2508.13938/) (2025)
- [ProJudge: A Multi-Modal Multi-Discipline Benchmark and Instruction-Tuning Dataset for MLLM-based Process Judges](https://mlsanity.com/paper/2503.06553/) (2025)
- [Towards Natural Language-Based Document Image Retrieval: New Dataset and Benchmark](https://mlsanity.com/paper/2512.20174/) (2025)
- [mmJEE-Eval: A Bilingual Multimodal Benchmark for Evaluating Scientific Reasoning in Vision-Language Models](https://mlsanity.com/paper/2511.09339/) (2025)
- [MATRIX: A Multimodal Benchmark and Post-Training Framework for Materials Science](https://mlsanity.com/paper/2602.00376/) (2026)
- [Defining the boundaries: challenges and advances in identifying cells in microscopy images](https://mlsanity.com/paper/2311.08269/) (2023)

## Abstract (arXiv)

Materials phase diagrams are a core knowledge representation in materials science, encoding temperature,composition, phase stability, and phase transformation pathways, with their full understanding requiring thermodynamic mechanism analysis and scientific reasoning. Although VLMs have shown promise in scientific image understanding, their systematic evaluation on such logically complex images demanding deep mechanistic interpretation remains limited, and phase diagrams provide a challenging testbed for this purpose. We introduce MatPhaseBench, a high-quality, high-reliability benchmark for complex scientific image understanding, focused on materials phase diagrams. MatPhaseBench is constructed from 3681 papers in classical materials science journals, from which 200 high-quality diagram-text pairs were selected, covering 189 material systems and 70 elements. The benchmark has three key features: (1)targeting complex scientific image understanding-it moves beyond simple objective tests to open-ended tasks requiring deep comprehension; (2)comprehensive image-text alignment-semantic information associated with images is fully preserved during literature mining and matching; (3) high-quality human-supervised text acquisition-all descriptions undergo strict manual validation. Experimental results show that current VLMs remain substantially behind expert-level understanding: they are largely limited to surface visual perception, lack deep reasoning grounded in thermodynamic mechanisms, have limited domain awareness and expert analytical experience, and perform poorly in distinguishing fine-grained differences in composite or multi-diagram settings. Overall, MatPhaseBench constitutes a challenging research-grade benchmark, providing a foundational platform for complex scientific image understanding, phase diagram analysis, and trustworthy multi-modal AI in science.