OmniMatBench: A Human-Calibrated Multimodal Reasoning Benchmark Across 19 Materials Science Subfields
On the map: LLM reasoning → Clinical and biomedical applications
Abstract (arXiv)
As multimodal language models play an increasingly important role in scientific research, materials science offers a critical testbed due to its interdisciplinary, multimodal, and application-driven nature. However, existing materials benchmarks mainly focus on property prediction, knowledge QA, or characterization understanding, leaving the broader reasoning process from materials knowledge to application underexplored. To fill this gap, we present OmniMatBench, a human-calibrated multimodal reasoning benchmark for materials science. OmniMatBench contains 3,171 expert-curated QA and calculation problems across 19 materials-science subfields, spanning fundamental materials knowledge, structural and engineering materials, materials processing and manufacturing, and functional and applied materials. We evaluate 13 open-source and closed-source MLLMs and find that the best model achieves only a 0.372 overall score, revealing a substantial gap in current materials-science reasoning. Further analysis shows strong variation across subfields, fixed reasoning heuristics, uneven materials knowledge, and limited high-level knowledge application under formula-, retrieval-, and code-assisted settings. OmniMatBench provides crucial insights into the capabilities and limitations of current MLLMs and establishes a foundation for reliable AI assistants in materials-science research.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning. Builds on 13 of its 31 references that are in the map. First posted on arXiv 28 May 2026.
Builds on
13 of 31 references.
Closest ancestors · 4
Same direction, newest first
- 2026MATRIX: A Multimodal Benchmark and Post-Training Framework for Materials Science30 Jan
- 2025L^2M^3OF: A Large Language Multimodal Model for Metal-Organic Frameworks23 Oct
- 2025MatSciBench: Benchmarking the Reasoning Ability of Large Language Models in Materials Science14 Oct
- 2025MSQA: Benchmarking LLMs on Graduate-Level Materials Science Reasoning and Knowledge29 May
Other references · 9
- 2024GPT-4o System Cardcited by 4,467
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
- 2025Qwen3-VL Technical Reportcited by 2,534
- 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
- 2026The Llama 4 Herd: Architecture, Training, Evaluation, and Deployment Notescited by 25
- 2026Intern-S1-Pro: Scientific Multimodal Foundation Model at Trillion Scalecited by 25
- 2025Seeing Beyond Words: MatVQA for Challenging Visual-Scientific Reasoning in Materials Sciencecited by 4
- 2025MatQnA: A Benchmark Dataset for Multi-modal Large Language Models in Materials Characterization and Analysiscited by 3
- 2026PolyReal: A Benchmark for Real-World Polymer Science Workflowscited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Physics as the label for measuring and correcting materials reasoning in multimodal models
- 2026MaterialFigBENCH: benchmark dataset with figures for evaluating college-level materials science problem-solving abilities of multimodal large language models
- 2024HoneyComb: A Flexible LLM-Based Agent System for Materials Science
- 2024Are LLMs Ready for Real-World Materials Discovery?
- 2025AtomWorld: A Benchmark for Evaluating Spatial Reasoning in Large Language Models on Crystalline Materials
- 2025Language-Native Materials Processing Design by Lightly Structured Text Database and Reasoning Large Language Model
- 2026SCALAR: Quantifying Structural Hallucination, Consistency, and Reasoning Gaps in Materials Foundation Models
- 2026ARIA: A Causal-Aware Framework for Rescuing LLM Reasoning in Trustworthy Materials Discovery