# SUPERChem: A Multimodal Reasoning Benchmark in Chemistry

arXiv: https://arxiv.org/abs/2512.01274 · HTML: https://mlsanity.com/paper/2512.01274/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 21 of its 28 references that are in the map.
- First posted on arXiv 1 Dec 2025.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)

## Builds on (most cited first, 21 of 21)

- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Humanity's Last Exam](https://mlsanity.com/paper/2501.14249/) (2025)
- [SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models](https://mlsanity.com/paper/2307.10635/) (2023)
- [What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks](https://mlsanity.com/paper/2305.18365/) (2023)
- [SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research](https://mlsanity.com/paper/2308.13149/) (2023)
- [SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines](https://mlsanity.com/paper/2502.14739/) (2025)
- [OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AI](https://mlsanity.com/paper/2406.12753/) (2024)
- [LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models](https://mlsanity.com/paper/2504.10415/) (2025)
- [PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Models](https://mlsanity.com/paper/2504.16074/) (2025)
- [Intern-S1: A Scientific Multimodal Foundation Model](https://mlsanity.com/paper/2508.15763/) (2025)
- [Beyond Chemical QA: Evaluating LLM's Chemical Reasoning with Modular Chemical Operations](https://mlsanity.com/paper/2505.21318/) (2025)
- [Benchmarking Retrieval-Augmented Generation for Chemistry](https://mlsanity.com/paper/2505.07671/) (2025)
- [ChemEval: A Comprehensive Multi-Level Chemical Evaluation for Large Language Models](https://mlsanity.com/paper/2409.13989/) (2024)
- [Benchmarking Large Language Models for Molecule Prediction Tasks](https://mlsanity.com/paper/2403.05075/) (2024)
- [Assessing the Chemical Intelligence of Large Language Models](https://mlsanity.com/paper/2505.07735/) (2025)
- [ChemSafetyBench: Benchmarking LLM Safety on Chemistry Domain](https://mlsanity.com/paper/2411.16736/) (2024)
- [Benchmarking Multimodal LLMs on Recognition and Understanding over Chemical Tables](https://mlsanity.com/paper/2506.11375/) (2025)
- [FGBench: A Dataset and Benchmark for Molecular Property Reasoning at Functional Group-Level in Large Language Models](https://mlsanity.com/paper/2508.01055/) (2025)
- [QCBench: Evaluating Large Language Models on Domain-Specific Quantitative Chemistry](https://mlsanity.com/paper/2508.01670/) (2025)

## Built on by (most cited first, 9 of 9)

- [Innovator-VL: A Multimodal Large Language Model for Scientific Discovery](https://mlsanity.com/paper/2601.19325/) (2026)
- [When Single Answer Is Not Enough: Rethinking Single-Step Retrosynthesis Benchmarks for LLMs](https://mlsanity.com/paper/2602.03554/) (2026)
- [From Answers to States: Verifiable Process-Level Evaluation of Chemical Reasoning in Large Language Models](https://mlsanity.com/paper/2606.03660/) (2026)
- [SynCraft: Guiding Large Language Models to Predict Edit Sequences for Molecular Synthesizability Optimization](https://mlsanity.com/paper/2512.20333/) (2025)
- [A3-Bench: Benchmarking Memory-Driven Scientific Reasoning via Anchor and Attractor Activation](https://mlsanity.com/paper/2601.09274/) (2026)
- [A Scientific Reasoning Model for Organic Synthesis Procedure Generation](https://mlsanity.com/paper/2512.13668/) (2025)
- [Cross-Domain Hybrid OPD for Generalizable Search Agents](https://mlsanity.com/paper/2608.02101/) (2026)
- [ChemExplain: A Multimodal Benchmark for Explanatory Chemical Reasoning](https://mlsanity.com/paper/2610.17183/) (2026)
- [ChemArgus: Automated Scientific Diagnostics for Lab-Grade Multimodal Chemistry Reasoning: A Frontier Benchmark](https://mlsanity.com/paper/2610.20012/) (2026)

## Abstract (arXiv)

Current benchmarks for evaluating the chemical reasoning capabilities of Large Language Models (LLMs) are limited by oversimplified tasks, lack of process-level evaluation, and misalignment with expert-level chemistry skills. To address these issues, we introduce SUPERChem, a benchmark of 500 expert-curated reasoning-intensive chemistry problems, covering diverse subfields and provided in both multimodal and text-only formats. Original content and an iterative curation pipeline eliminate flawed items and mitigate data contamination. Each problem is paired with an expert-authored solution path, enabling Reasoning Path Fidelity (RPF) scoring to evaluate reasoning quality beyond final-answer accuracy. Evaluations against a human baseline of 40.3% accuracy show that even the best-performing model, GPT-5 (High), reaches only 38.5%, followed closely by Gemini 2.5 Pro (37.9%) and DeepSeek-V3.1-Think (37.3%). SUPERChem elicits multi-step, multimodal reasoning, reveals model-dependent effects of visual information, and distinguishes high-fidelity reasoners from heuristic ones. By providing a challenging benchmark and a reliable evaluation framework, SUPERChem aims to facilitate the advancement of LLMs toward expert-level chemical intelligence. The dataset of the benchmark is available at https://huggingface.co/datasets/ZehuaZhao/SUPERChem.