4DCodeBench: Benchmarking Agents on Inverse Graphics of Dynamic Scenes
Not on the map. Why?
Abstract (arXiv)
We introduce 4DCodeBench, a benchmark for 4D inverse graphics through code generation, in which agents reconstruct dynamic scenes from video as executable graphics programs. To accomplish this, agents must translate visual observations into compact representations of scene structure and dynamics, by implementing abstractions such as physical simulations to reproduce complex behavior. To evaluate this capability, we curate a set of real-world videos and construct synthetic scenes spanning diverse physical phenomena, including deformation, fluid flow, and fracture. We perform extensive benchmarking of frontier models, finding that strong static reconstruction capabilities do not yet translate into reliable reconstruction of complex dynamics. 4DCodeBench provides a testbed for tracking progress toward agents that can interpret the dynamics of the world through code. Our benchmark is available at https://github.com/4DCodeBench/4DCodeBench
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 48 of its 78 references that are in the map. First posted on arXiv 2 Oct 2026.
Builds on
48 of 78 references.
References · 48
- 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,415
- 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,495
- 2025DINOv3cited by 1,414
- 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
- 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
- 2025AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systemscited by 405
- 2019DiffTaichi: Differentiable Programming for Physical Simulationcited by 258
- 2023Uni3D: Exploring Unified 3D Representation at Scalecited by 237
- 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 228
- 2025Video Depth Anything: Consistent Depth Estimation for Super-Long Videoscited by 207
- 2025Do generative video models understand physical principles?cited by 161
- 2025PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understandingcited by 160
36 more
- 2024ALOHA Unleashed: A Simple Recipe for Robot Dexteritycited by 134
- 2024CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videoscited by 128
- 2025WISA: World Simulator Assistant for Physics-Aware Text-to-Video Generationcited by 90
- 2023RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse Toolscited by 76
- 2022RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networkscited by 72
- 2023Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Propertiescited by 54
- 2025IntPhys 2: Benchmarking Intuitive Physics Understanding In Complex Synthetic Environmentscited by 52
- 2026Vision-as-Inverse-Graphics Agent via Interleaved Multimodal Reasoningcited by 49
- 2023Neurosymbolic Models for Computer Graphicscited by 28
- 2024Re-Thinking Inverse Graphics With Large Language Modelscited by 23
- 2025QuantiPhy: A Quantitative Benchmark Evaluating Physical Reasoning Abilities of Vision-Language Modelscited by 20
- 2026Code2Worlds: Empowering Coding LLMs for 4D World Generationcited by 19
- 20263DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Codecited by 15
- 2026Code as Worlds: Agentic Discovery of Executable World Representations for Physical Reasoningcited by 14
- 2026TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignmentcited by 13
- 2026T-Rex: Tactile-Reactive Dexterous Manipulationcited by 13
- 2025Towards Visual Discrimination and Reasoning of Real-World Physical Dynamics: Physics-Grounded Anomaly Detectioncited by 10
- 2026PhysCodeBench: Benchmarking Physics-Aware Symbolic Simulation of 3D Scenes via Self-Corrective Multi-Agent Refinementcited by 10
- 2026Thinking in Blender: Staged Executable Inverse Graphics with Vision-Language Modelscited by 10
- 2026Scalable Behavior Cloning with Open Data, Training, and Evaluationcited by 10
- 2023Robo360: A 3D Omnispective Multi-Material Robotic Manipulation Datasetcited by 9
- 2026SimWorlds: A Multi-Agent System for Dynamic 3D Scene Creationcited by 8
- 2026MoGe-3: Fine-Detail Monocular Geometry Estimation with Self-Guided Sparse Volumetric Refinementcited by 8
- 2023GIPC: Fast and stable Gauss-Newton optimization of IPC barrier energycited by 7
- 2026VisPhyWorld: Probing Physical Reasoning via Code-Driven Video Reconstructioncited by 7
- 2024StiffGIPC: Advancing GPU IPC for stiff affine-deformable simulationcited by 6
- 2026SceneActBench: Can Agents Act on the 3D Scenes They See?cited by 6
- 2026BVB: Benchmarking Agentic Video Understanding via Programmatic Reconstruction in Blendercited by 6
- 2026ManipArena: Comprehensive Real-world Evaluation of Reasoning-Oriented Generalist Robot Manipulationcited by 5
- 2026MPMWorlds: Material-Point-Method Simulations for Inferring and Extrapolating Physical Dynamicscited by 5
- 2019Hierarchical Optimization Time Integration for CFL-rate MPM Steppingcited by 3
- 2026Deltaynamics: Language-Based Representation for Inferring Rigid-Body Dynamics From Videoscited by 3
- 2026GEOPHYS: The Geometry of Physical Plausibilitycited by 3
- 2026Learning Dynamic Rope Manipulation Using Task-Level Iterative Learning Controlcited by 2
- 2024CK-MPM: A Compact-Kernel Material Point Methodcited by 1
- 2026MPM Lite: Linear Kernels and Integration without Particlescited by 1
Built on by
Citing papers per year: 2026: 3.
3 papers.