# FeynmanBench: Benchmarking Multimodal LLMs on Diagrammatic Physics Reasoning

arXiv: https://arxiv.org/abs/2604.03893 · HTML: https://mlsanity.com/paper/2604.03893/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 31 of its 60 references that are in the map.
- First posted on arXiv 4 Apr 2026.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)
- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 30 of 31)

- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Emerging Properties in Unified Multimodal Pretraining](https://mlsanity.com/paper/2505.14683/) (2025)
- [TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation](https://mlsanity.com/paper/2412.03069/) (2024)
- [MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI](https://mlsanity.com/paper/2404.16006/) (2024)
- [TheoremQA: A Theorem-driven Question Answering dataset](https://mlsanity.com/paper/2305.12524/) (2023)
- [VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models](https://mlsanity.com/paper/2504.15279/) (2025)
- [Harmonizing Visual Representations for Unified Multimodal Understanding and Generation](https://mlsanity.com/paper/2503.21979/) (2025)
- [Masked Vision and Language Modeling for Multi-modal Representation Learning](https://mlsanity.com/paper/2208.02131/) (2022)
- [Unified Autoregressive Visual Generation and Understanding with Continuous Tokens](https://mlsanity.com/paper/2503.13436/) (2025)
- [Multimodal Latent Language Modeling with Next-Token Diffusion](https://mlsanity.com/paper/2412.08635/) (2024)
- [PhyX: Does Your Model Have the "Wits" for Physical Reasoning?](https://mlsanity.com/paper/2505.15929/) (2025)
- [UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling](https://mlsanity.com/paper/2408.04810/) (2024)
- [Physics: Benchmarking Foundation Models on University-Level Physics Problem Solving](https://mlsanity.com/paper/2503.21821/) (2025)
- [LLMPhy: Parameter-Identifiable Physical Reasoning Combining Large Language Models and Physics Engines](https://mlsanity.com/paper/2411.08027/) (2024)
- [ContPhy: Continuum Physical Concept Learning and Reasoning from Videos](https://mlsanity.com/paper/2402.06119/) (2024)
- [PhysUniBench: A Multi-Modal Physics Reasoning Benchmark at Undergraduate Level](https://mlsanity.com/paper/2506.17667/) (2025)
- [MM-PhyQA: Multimodal Physics Question-Answering With Multi-Image CoT Prompting](https://mlsanity.com/paper/2404.08704/) (2024)
- [Structures of Neural Network Effective Theories](https://mlsanity.com/paper/2305.02334/) (2023)
- [Understanding Diffusion Models by Feynman's Path Integral](https://mlsanity.com/paper/2403.11262/) (2024)
- [MAPS: Advancing Multi-Modal Reasoning in Expert-Level Physical Science](https://mlsanity.com/paper/2501.10768/) (2025)
- [Unified Multimodal Understanding via Byte-Pair Visual Encoding](https://mlsanity.com/paper/2506.23639/) (2025)
- [Discrete Tokenization for Multimodal LLMs: A Comprehensive Survey](https://mlsanity.com/paper/2507.22920/) (2025)
- [Refining Integration-by-Parts Reduction of Feynman Integrals with Machine Learning](https://mlsanity.com/paper/2502.05121/) (2025)
- [Image Tokens Matter: Mitigating Hallucination in Discrete Tokenizer-based Large Vision-Language Models via Latent Editing](https://mlsanity.com/paper/2505.21547/) (2025)
- [Neural network representation of quantum systems](https://mlsanity.com/paper/2403.11420/) (2024)
- [Finite-Width Neural Tangent Kernels from Feynman Diagrams](https://mlsanity.com/paper/2508.11522/) (2025)
- [Uncovering Singularities in Feynman Integrals via Machine Learning](https://mlsanity.com/paper/2510.10099/) (2025)

## Abstract (arXiv)

Current multimodal benchmarks for scientific reasoning primarily evaluate local information extraction -- models recognize symbols and values and then perform textual inference. They do not assess whether models can reason over the global structural properties of formal diagrams, such as topology, conservation constraints, and the consistent mapping between visual patterns and algebraic expressions. We introduce FeynmanBench, a benchmark of over 2,000 tasks centered on Feynman diagrams spanning the electromagnetic, weak, and strong interactions of the Standard Model. Each instance couples a diagram image with minimal textual conventions and requires models to recover the full physical content -- vertex inventory, propagator types, topological connectivity, momentum routing, and the complete scattering amplitude. An automated generation and verification pipeline produces the diagrams, annotations, and reference answers under standardized rules. Evaluating 19 state-of-the-art multimodal LLMs, we find a consistent failure pattern: models achieve 70--95% on local recognition (vertex and propagator identification) but collapse to 13--17% on topological reconstruction (CP3), and near zero on full algebraic derivation (CP5). FeynmanBench offers a controlled testbed for multimodal reasoning over formal scientific diagrams and highlights fundamental limitations of current architectures in topology-sensitive scientific reasoning.