mlsanityData to 14 Oct 2026

SeePhys: Does Seeing Help Thinking? -- Benchmarking Vision-Based Physics Reasoning

arXiv 2505.19099 ↗First posted 25 May 2025cs.AI physics.ed-ph physics.pop-phcited by 34

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

We present SeePhys, a large-scale multimodal benchmark for LLM reasoning grounded in physics questions ranging from middle school to PhD qualifying exams. The benchmark covers 7 fundamental domains spanning the physics discipline, incorporating 21 categories of highly heterogeneous diagrams. In contrast to prior works where visual elements mainly serve auxiliary purposes, our benchmark features a substantial proportion of vision-essential problems (75%) that mandate visual information extraction for correct solutions. Through extensive evaluation, we observe that even the most advanced visual reasoning models (e.g., Gemini-2.5-pro and o4-mini) achieve sub-60% accuracy on our benchmark. These results reveal fundamental challenges in current large language models' visual understanding capabilities, particularly in: (i) establishing rigorous coupling between diagram interpretation and physics reasoning, and (ii) overcoming their persistent reliance on textual cues as cognitive shortcuts.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 33 of its 41 references that are in the map. First posted on arXiv 25 May 2025.

Builds on

33 of 41 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Models22 Apr
  2. 2025Theoretical Physics Benchmark (TPBench) -- a Dataset and Study of AI Reasoning Capabilities in Theoretical Physics19 Feb
  3. 2025PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoning17 Feb
  4. 2025UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models1 Feb
  5. 2022UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expression6 Dec

Other references · 28

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  3. 2025Qwen2.5-VL Technical Reportcited by 5,587
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  8. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  11. 2025Gemma 3 Technical Reportcited by 1,824
  12. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
16 more
  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  2. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  3. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  4. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 526
  5. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 521
  6. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
  7. 2025Humanity's Last Examcited by 491
  8. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 284
  9. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 267
  10. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 256
  11. 2021GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoningcited by 228
  12. 2021Physion: Evaluating Physical Prediction from Vision in Humans and Machinescited by 153
  13. 2023SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Researchcited by 115
  14. 2023ARB: Advanced Reasoning Benchmark for Large Language Modelscited by 60
  15. 2025Physics: Benchmarking Foundation Models on University-Level Physics Problem Solvingcited by 31
  16. 2025Can Atomic Step Decomposition Enhance the Self-structured Reasoning of Multimodal Large Models?cited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 22.

34 papers.

Sort:

By area:

  1. 2025PhysUniBench: A Multi-Modal Physics Reasoning Benchmark at Undergraduate Levelcited by 16
  2. 2025DeepPHY: Benchmarking Agentic VLMs on Physical Reasoningcited by 14
  3. 2025A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trendscited by 8
  4. 2026Scientific Image Synthesis: Benchmarking, Methodologies, and Downstream Utilitycited by 8
  5. 2025PhysicsMinions: Winning Gold Medals in the Latest Physics Olympiads with a Coevolutionary Multimodal Multi-Agent Systemcited by 7
  6. 2026Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understandingcited by 7
  7. 2025MORSE-500: A Programmatically Controllable Video Benchmark to Stress-Test Multimodal Reasoningcited by 6
  8. 2025VisioMath: Benchmarking Figure-based Mathematical Reasoning in LMMscited by 6
  9. 2025Multimodal Reasoning for Science: Technical Report and 1st Place Solution to the ICML 2025 SeePhys Challengecited by 6
  10. 2026Accordion-Thinking: Self-Regulated Step Summaries for Efficient and Readable LLM Reasoningcited by 6
  11. 2026OmniFysics: Towards Physical Intelligence Evolution via Omni-Modal Signal Processing and Network Optimizationcited by 5
  12. 2025PRISM-Physics: Causal DAG-Based Process Evaluation for Physics Reasoningcited by 4
22 more
  1. 2025CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal Reasoningcited by 4
  2. 2025CircuitSense: A Hierarchical MLLM Benchmark Bridging Visual Comprehension and Symbolic Reasoning in Engineering Design Processcited by 3
  3. 2025Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AIcited by 3
  4. 2026Stargazer: A Scalable Model-Fitting Benchmark Environment for AI Agents under Astrophysical Constraintscited by 3
  5. 2026Seek-and-Solve: Benchmarking MLLMs for Visual Clue-Driven Reasoning in Daily Scenarioscited by 2
  6. 2025GeoThought: A Dataset for Enhancing Mathematical Geometry Reasoning in Vision-Language Modelscited by 1
  7. 2026SketchThinker-R1: Towards Efficient Sketch-Style Reasoning in Large Multimodal Modelscited by 1
  8. 2026Order from Chaos: Physical World Understanding from Glitchy Gameplay Videoscited by 1
  9. 2026SeePhys Pro: Diagnosing Modality Transfer and Blind-Training Effects in Multimodal RLVR for Physics Reasoningcited by 1
  10. 2026BilliardPhys-Bench: Benchmarking Physical Reasoning and Visual Dynamics of Multimodal LLMscited by 1
  11. 2026Enginuity: A Dataset and Benchmark for Vision-Language Understanding of Engineering Diagramscited by 1
  12. 2026Discovering High Level Patterns from Simulation Tracescited by 0
  13. 2026RADAR: Revealing Asymmetric Development of Abilities in MLLM Pre-trainingcited by 0
  14. 2026SPM-Bench: Benchmarking Large Language Models for Scanning Probe Microscopycited by 0
  15. 2026Do VLMs Reason Like Engineers? A Benchmark and a Stage-wise Evaluationcited by 0
  16. 2026OmniPhys: A Unified Multimodal Benchmark for Physics Understanding and Generation from Chinese Educational Corporacited by 0
  17. 2026Circuit-MLLM: Topological Logic-Guided Latent-Space Visual Reasoning for Circuit Schematic Understandingcited by 0
  18. 2026Lost in Perception: Isolating Perceptual and Reasoning Failures in Multimodal Physics and Geometry Reasoningcited by 0
  19. 2026EngIntervene: Benchmarking Multimodal Engineering State Understanding and Design Intervention Reasoningcited by 0
  20. 2026PhysAlign: A Benchmark for Evidence-Grounded Role Alignment in Multimodal Physics Reasoningcited by 0
  21. 2026PhysFieldBench: Can Multimodal Models Understand Physical Fields?cited by 0
  22. 2026Adaptive Look, Flexible Thought: Learning How and How Long to Think in Multimodal Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Multi-Physics: A Comprehensive Benchmark for Multimodal LLMs Reasoning on Chinese Multi-Subject Physics Problems
  2. 2025Interpretable Physics Reasoning and Performance Taxonomy in Vision-Language Models
  3. 2026P1-VL: Bridging Visual Perception and Scientific Reasoning in Physics Olympiads
  4. 2025MathSight: A Benchmark Exploring Have Vision-Language Models Really Seen in University-Level Mathematical Reasoning?
  5. 2024VisScience: An Extensive Benchmark for Evaluating K12 Educational Multi-modal Scientific Reasoning
  6. 2025PhyX: Does Your Model Have the "Wits" for Physical Reasoning?
  7. 2025LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMs
  8. 2024Physics Context Builders: A Modular Framework for Physical Reasoning in Vision-Language Models