# SeePhys: Does Seeing Help Thinking? -- Benchmarking Vision-Based Physics Reasoning

arXiv: https://arxiv.org/abs/2505.19099 · HTML: https://mlsanity.com/paper/2505.19099/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 33 of its 41 references that are in the map.
- First posted on arXiv 25 May 2025.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)

## Builds on (most cited first, 30 of 33)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?](https://mlsanity.com/paper/2403.14624/) (2024)
- [Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset](https://mlsanity.com/paper/2402.14804/) (2024)
- [Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs](https://mlsanity.com/paper/2503.01743/) (2025)
- [MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark](https://mlsanity.com/paper/2409.02813/) (2024)
- [Humanity's Last Exam](https://mlsanity.com/paper/2501.14249/) (2025)
- [Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models](https://mlsanity.com/paper/2410.07985/) (2024)
- [IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning](https://mlsanity.com/paper/2110.13214/) (2021)
- [SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models](https://mlsanity.com/paper/2307.10635/) (2023)
- [GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning](https://mlsanity.com/paper/2105.14517/) (2021)
- [Physion: Evaluating Physical Prediction from Vision in Humans and Machines](https://mlsanity.com/paper/2106.08261/) (2021)
- [UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expression](https://mlsanity.com/paper/2212.02746/) (2022)
- [SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research](https://mlsanity.com/paper/2308.13149/) (2023)
- [PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Models](https://mlsanity.com/paper/2504.16074/) (2025)
- [ARB: Advanced Reasoning Benchmark for Large Language Models](https://mlsanity.com/paper/2307.13692/) (2023)
- [PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoning](https://mlsanity.com/paper/2502.12054/) (2025)
- [UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models](https://mlsanity.com/paper/2502.00334/) (2025)

## Built on by (most cited first, 30 of 34)

- [PhysUniBench: A Multi-Modal Physics Reasoning Benchmark at Undergraduate Level](https://mlsanity.com/paper/2506.17667/) (2025)
- [DeepPHY: Benchmarking Agentic VLMs on Physical Reasoning](https://mlsanity.com/paper/2508.05405/) (2025)
- [A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trends](https://mlsanity.com/paper/2507.09861/) (2025)
- [Scientific Image Synthesis: Benchmarking, Methodologies, and Downstream Utility](https://mlsanity.com/paper/2601.17027/) (2026)
- [PhysicsMinions: Winning Gold Medals in the Latest Physics Olympiads with a Coevolutionary Multimodal Multi-Agent System](https://mlsanity.com/paper/2509.24855/) (2025)
- [Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understanding](https://mlsanity.com/paper/2603.18472/) (2026)
- [MORSE-500: A Programmatically Controllable Video Benchmark to Stress-Test Multimodal Reasoning](https://mlsanity.com/paper/2506.05523/) (2025)
- [VisioMath: Benchmarking Figure-based Mathematical Reasoning in LMMs](https://mlsanity.com/paper/2506.06727/) (2025)
- [Multimodal Reasoning for Science: Technical Report and 1st Place Solution to the ICML 2025 SeePhys Challenge](https://mlsanity.com/paper/2509.06079/) (2025)
- [Accordion-Thinking: Self-Regulated Step Summaries for Efficient and Readable LLM Reasoning](https://mlsanity.com/paper/2602.03249/) (2026)
- [OmniFysics: Towards Physical Intelligence Evolution via Omni-Modal Signal Processing and Network Optimization](https://mlsanity.com/paper/2602.07064/) (2026)
- [PRISM-Physics: Causal DAG-Based Process Evaluation for Physics Reasoning](https://mlsanity.com/paper/2510.03185/) (2025)
- [CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal Reasoning](https://mlsanity.com/paper/2512.19554/) (2025)
- [CircuitSense: A Hierarchical MLLM Benchmark Bridging Visual Comprehension and Symbolic Reasoning in Engineering Design Process](https://mlsanity.com/paper/2509.22339/) (2025)
- [Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AI](https://mlsanity.com/paper/2510.04978/) (2025)
- [Stargazer: A Scalable Model-Fitting Benchmark Environment for AI Agents under Astrophysical Constraints](https://mlsanity.com/paper/2604.15664/) (2026)
- [Seek-and-Solve: Benchmarking MLLMs for Visual Clue-Driven Reasoning in Daily Scenarios](https://mlsanity.com/paper/2604.14041/) (2026)
- [GeoThought: A Dataset for Enhancing Mathematical Geometry Reasoning in Vision-Language Models](https://mlsanity.com/paper/2510.21881/) (2025)
- [SketchThinker-R1: Towards Efficient Sketch-Style Reasoning in Large Multimodal Models](https://mlsanity.com/paper/2601.02825/) (2026)
- [Order from Chaos: Physical World Understanding from Glitchy Gameplay Videos](https://mlsanity.com/paper/2601.16471/) (2026)
- [SeePhys Pro: Diagnosing Modality Transfer and Blind-Training Effects in Multimodal RLVR for Physics Reasoning](https://mlsanity.com/paper/2605.09266/) (2026)
- [BilliardPhys-Bench: Benchmarking Physical Reasoning and Visual Dynamics of Multimodal LLMs](https://mlsanity.com/paper/2605.30900/) (2026)
- [Enginuity: A Dataset and Benchmark for Vision-Language Understanding of Engineering Diagrams](https://mlsanity.com/paper/2606.03410/) (2026)
- [Discovering High Level Patterns from Simulation Traces](https://mlsanity.com/paper/2602.10009/) (2026)
- [RADAR: Revealing Asymmetric Development of Abilities in MLLM Pre-training](https://mlsanity.com/paper/2602.12892/) (2026)
- [SPM-Bench: Benchmarking Large Language Models for Scanning Probe Microscopy](https://mlsanity.com/paper/2602.22971/) (2026)
- [Do VLMs Reason Like Engineers? A Benchmark and a Stage-wise Evaluation](https://mlsanity.com/paper/2606.10833/) (2026)
- [OmniPhys: A Unified Multimodal Benchmark for Physics Understanding and Generation from Chinese Educational Corpora](https://mlsanity.com/paper/2608.25398/) (2026)
- [Circuit-MLLM: Topological Logic-Guided Latent-Space Visual Reasoning for Circuit Schematic Understanding](https://mlsanity.com/paper/2609.15668/) (2026)
- [Lost in Perception: Isolating Perceptual and Reasoning Failures in Multimodal Physics and Geometry Reasoning](https://mlsanity.com/paper/2609.18991/) (2026)

## Abstract (arXiv)

We present SeePhys, a large-scale multimodal benchmark for LLM reasoning grounded in physics questions ranging from middle school to PhD qualifying exams. The benchmark covers 7 fundamental domains spanning the physics discipline, incorporating 21 categories of highly heterogeneous diagrams. In contrast to prior works where visual elements mainly serve auxiliary purposes, our benchmark features a substantial proportion of vision-essential problems (75%) that mandate visual information extraction for correct solutions. Through extensive evaluation, we observe that even the most advanced visual reasoning models (e.g., Gemini-2.5-pro and o4-mini) achieve sub-60% accuracy on our benchmark. These results reveal fundamental challenges in current large language models' visual understanding capabilities, particularly in: (i) establishing rigorous coupling between diagram interpretation and physics reasoning, and (ii) overcoming their persistent reliance on textual cues as cognitive shortcuts.