Order from Chaos: Physical World Understanding from Glitchy Gameplay Videos
Not on the map. Why?
Abstract (arXiv)
Understanding the physical world, including object dynamics, material properties, and causal interactions, remains a core challenge in artificial intelligence. Although recent multi-modal large language models (MLLMs) have demonstrated impressive general reasoning capabilities, they still fall short of achieving human-level understanding of physical principles. Existing datasets for physical reasoning either rely on real-world videos, which incur high annotation costs, or on synthetic simulations, which suffer from limited realism and diversity. In this paper, we propose a novel paradigm that leverages glitches in gameplay videos, referring to visual anomalies that violate predefined physical laws, as a rich and scalable supervision source for physical world understanding. We introduce PhysGame, an meta information guided instruction-tuning dataset containing 140,057 glitch-centric question-answer pairs across five physical domains and sixteen fine-grained categories. To ensure data accuracy, we design a prompting strategy that utilizes gameplay metadata such as titles and descriptions to guide high-quality QA generation. Complementing PhysGame, we construct GameBench, an expert-annotated benchmark with 880 glitch-identified gameplay videos designed to evaluate physical reasoning capabilities. Extensive experiments show that PhysGame significantly enhances both Game2Real transferability, improving the real world physical reasoning performance of Qwen2.5VL by 2.5% on PhysBench, and Game2General transferability, yielding a 1.9% gain on the MVBench benchmark. Moreover, PhysGame-tuned models achieve a 3.7% absolute improvement on GameBench, demonstrating enhanced robustness in detecting physical implausibilities. These results indicate that learning from gameplay anomalies offers a scalable and effective pathway toward advancing physical world understanding in multimodal intelligence.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 75 of its 91 references that are in the map. First posted on arXiv 23 Jan 2026.
Builds on
75 of 91 references.
References · 75
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2023GPT-4 Technical Reportcited by 18,396
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2025Qwen2.5-VL Technical Reportcited by 5,587
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
- 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
- 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
- 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
- 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
- 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
38 more
- 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
- 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
- 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
- 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
- 2023VILA: On Pre-training for Visual Language Modelscited by 692
- 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 665
- 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 598
- 2024Long Context Transfer from Language to Visioncited by 515
- 2016Interaction Networks for Learning about Objects, Relations and Physicscited by 509
- 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 484
- 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 431
- 2018VirtualHome: Simulating Household Activities via Programscited by 415
- 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 388
- 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
- 2019On the Utility of Learning about Humans for Human-AI Coordinationcited by 349
- 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 322
- 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulationcited by 250
- 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 228
- 2024NVILA: Efficient Frontier Visual Language Modelscited by 209
- 2024Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generationcited by 174
- 2025PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understandingcited by 160
- 2016Learning to Poke by Poking: Experiential Learning of Intuitive Physicscited by 154
- 2021Physion: Evaluating Physical Prediction from Vision in Humans and Machinescited by 153
- 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Rewardcited by 147
- 2025MMVU: Measuring Expert-Level Multi-Discipline Video Understandingcited by 137
- 2020Watch-And-Help: A Challenge for Social Perception and Human-AI Collaborationcited by 133
- 2024ST-LLM: Large Language Models Are Effective Temporal Learnerscited by 126
- 2016A Compositional Object-Based Approach to Learning Physical Dynamicscited by 119
- 2024A Survey on Large Language Model-Based Game Agentscited by 112
- 2023Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplationcited by 104
- 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecturecited by 102
- 2024Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videoscited by 98
- 2024Agent-Pro: Learning to Evolve via Policy-Level Reflection and Optimizationcited by 94
- 2024Apollo: An Exploration of Video Understanding in Large Multimodal Modelscited by 89
- 2023MindAgent: Emergent Gaming Interactioncited by 88
- 2023Large Language Models Play StarCraft II: Benchmarks and A Chain of Summarization Approachcited by 79
- 2023ChessGPT: Bridging Policy Learning and Language Modelingcited by 75
- 2021Chess as a Testbed for Language Model State Trackingcited by 72
50 most cited of 75 All 75 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2024PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videos
- 2026PhysEditWorld: A Large-Scale Dataset Toward Physics-Editable World Models
- 2026Asking the World: Generalist Physical Reasoning through Agentic World Modeling and Probing
- 2026PhyGround: Benchmarking Physical Reasoning in Generative World Models
- 2026Physion-Eval: Evaluating Physical Realism in Generated Video via Human Reasoning
- 2026PhysMind: From Video to Executable Worlds for Training-Free Physical Reasoning
- 2026PhysicsMind: Sim and Real Mechanics Benchmarking for Physical Reasoning and Prediction in Foundational VLMs and World Models
- 2026WorldBench: Benchmarking Physical Understanding of World Models by Isolating Physics Concepts