mlsanityData to 14 Oct 2026

PhySimCode: A Benchmark and Evaluation Method for Physics Video to Code Generation

arXiv 2610.15722 ↗First posted 11 Oct 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs also fits: Embodied and 3D vision-language models

Abstract (arXiv)

Multimodal large language models (MLLMs) are good at video understanding and code generation, but it remains unclear whether they can recover the physical laws underlying videos, a capability essential for converting videos into executable simulations and for providing structured physical supervision in robotics. Captioning motion video does not imply that a model has inferred its governing dynamics, estimated quantitative parameters, or can regenerate the observed scene from code. Existing video-physics benchmarks evaluate plausibility, property estimation, or future-state prediction, but offer limited insight into whether models can translate visual dynamics into runnable simulation code. We introduce PhySimCode, a benchmark for physics video-to-code generation: given only a raw video, a model must infer the relevant physics, recover parameter values, and generate runnable Python that reproduces the input video. The dataset contains 160.6K procedurally generated samples spanning 162 physics phenomena across analytic SciPy simulations and PyBullet-rendered rigid-body, articulated, fluid, and contact-rich scenes; each sample bundles the simulation code, physical parameters, rendered video, and a structured chain-of-thought covering observations, laws, ODEs, numerical methods, and validation checks. We evaluate ten state-of-the-art closed- and open-source MLLMs along physical-law correctness, simulation equivalence, code executability, and parameter recovery. Models reliably state physics laws (4.27-4.93/5) yet score only 1.51-2.47/5 on inferring them from video, and the best system (Claude Sonnet 4.6) recovers just 7.7% of ground-truth parameters within ± 20%, exposing a substantial video-to-simulation gap. The 2,430-video evaluation set is available at https://huggingface.co/datasets/SourajaKundu123/PhySimCode and the evaluation code at https://github.com/SourajaKundu/PhySimCode

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 50 references that are in the map. First posted on arXiv 11 Oct 2026.

Builds on

35 of 50 references.

References · 35

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  3. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
  4. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  5. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  6. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
  7. 2025Cosmos World Foundation Model Platform for Physical AIcited by 864
  8. 2016Interaction Networks for Learning about Objects, Relations and Physicscited by 509
  9. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 484
  10. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  11. 2023PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamicscited by 377
  12. 2019DiffTaichi: Differentiable Programming for Physical Simulationcited by 258
23 more
  1. 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 228
  2. 2025PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understandingcited by 160
  3. 2023PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identificationcited by 154
  4. 2021Physion: Evaluating Physical Prediction from Vision in Humans and Machinescited by 153
  5. 2024Reconstruction and Simulation of Elastic Objects with Spring-Mass 3D Gaussianscited by 97
  6. 2017Visual Interaction Networkscited by 95
  7. 2025ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generationcited by 92
  8. 2020NeuralSim: Augmenting Differentiable Simulators with Neural Networkscited by 82
  9. 2016Learning Physical Intuition of Block Towers by Examplecited by 68
  10. 2022ComPhy: Compositional Physical Reasoning of Objects and Events from Videoscited by 63
  11. 2025PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Modelscited by 61
  12. 2024GIC: Gaussian-Informed Continuum for Physical Property Identification and Simulationcited by 49
  13. 2019Physics-as-Inverse-Graphics: Unsupervised Physical Parameter Estimation from Videocited by 45
  14. 2024Neural Material Adaptor for Visual Grounding of Intrinsic Dynamicscited by 32
  15. 2024ContPhy: Continuum Physical Concept Learning and Reasoning from Videoscited by 27
  16. 2025VinciCoder: Unifying Multimodal Code Generation via Coarse-to-fine Visual Reinforcement Learningcited by 21
  17. 2019Visual Physics: Discovering Physical Laws from Videoscited by 10
  18. 2026PhysCodeBench: Benchmarking Physics-Aware Symbolic Simulation of 3D Scenes via Self-Corrective Multi-Agent Refinementcited by 10
  19. 2025MASIV: Toward Material-Agnostic System Identification from Videoscited by 9
  20. 2025Inferring Dynamic Physical Properties from Video Foundation Modelscited by 6
  21. 2026MM-ReCoder: Advancing Chart-to-Code Generation with Reinforcement Learning and Self-Correctioncited by 5
  22. 2026Solving Physics Olympiad via Reinforcement Learning on Physics Simulatorscited by 4
  23. 2026Training and Benchmarking Code Generation for Physics-Inspired Animationscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation
  2. 2025"PhyWorldBench": A Comprehensive Evaluation of Physical Realism in Text-to-Video Models
  3. 2026PhyGround: Benchmarking Physical Reasoning in Generative World Models
  4. 2026PhysicsLENS: Diagnosing Physical Property Blindness in Video Generation Models
  5. 2026Learning Explicit Physical Parameter Control and Benchmarking for Video Generation
  6. 2024PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation
  7. 2024PhysMotion: Physics-Grounded Dynamics From a Single Image
  8. 2026PhysInOne: Visual Physics Learning and Reasoning in One Suite