# Runtime Failure Hunting for Physics Engine Based Software Systems: How Far Can We Go?

arXiv: https://arxiv.org/abs/2507.22099 · HTML: https://mlsanity.com/paper/2507.22099/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 43 references that are in the map.
- First posted on arXiv 29 Jul 2025.

## Builds on (most cited first, 17 of 17)

- [AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles](https://mlsanity.com/paper/1705.05065/) (2017)
- [VideoPhy: Evaluating Physical Commonsense for Video Generation](https://mlsanity.com/paper/2406.03520/) (2024)
- [VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generation](https://mlsanity.com/paper/2406.15252/) (2024)
- [Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation](https://mlsanity.com/paper/2410.05363/) (2024)
- [MIST: Multiple Instance Self-Training Framework for Video Anomaly Detection](https://mlsanity.com/paper/2104.01633/) (2021)
- [Evaluation of Text-to-Video Generation Models: A Dynamics Perspective](https://mlsanity.com/paper/2407.01094/) (2024)
- [Holmes-VAD: Towards Unbiased and Explainable Video Anomaly Detection via Multi-modal LLM](https://mlsanity.com/paper/2406.12235/) (2024)
- [Augmenting Automated Game Testing with Deep Reinforcement Learning](https://mlsanity.com/paper/2103.15819/) (2021)
- [GlitchBench: Can large multimodal models detect video game glitches?](https://mlsanity.com/paper/2312.05291/) (2023)
- [CLIP meets GamePhysics: Towards bug identification in gameplay videos using zero-shot transfer learning](https://mlsanity.com/paper/2203.11096/) (2022)
- [GLIB: Towards Automated Test Oracle for Graphically-Rich Applications](https://mlsanity.com/paper/2106.10507/) (2021)
- [Automatically Detecting Visual Bugs in HTML5 <canvas> Games](https://mlsanity.com/paper/2208.02335/) (2022)
- [Modeling Software Quality in Virtual Reality Applications from User Feedback](https://mlsanity.com/paper/2308.06783/) (2023)
- [3D Software Synthesis Driven by Constraint-Expressive Intermediate Representation](https://mlsanity.com/paper/2507.18625/) (2025)
- [Less Cybersickness, Please: Demystifying and Detecting Stereoscopic Visual Inconsistencies in Virtual Reality Apps](https://mlsanity.com/paper/2406.09313/) (2024)
- [PHYFU: Fuzzing Modern Physics Simulation Engines](https://mlsanity.com/paper/2307.10818/) (2023)
- [XRZoo: A Large-Scale and Versatile Dataset of Extended Reality (XR) Applications](https://mlsanity.com/paper/2412.06759/) (2024)

## Abstract (arXiv)

Physics Engines (PEs) are fundamental software frameworks that simulate physical interactions in applications ranging from entertainment to safety-critical systems. Despite their importance, PEs suffer from physics failures, deviations from expected physical behaviors that can compromise software reliability, degrade user experience, and potentially cause critical failures in autonomous vehicles or medical robotics. Current testing approaches for PE-based software are inadequate, typically requiring white-box access and focusing on crash detection rather than semantically complex physics failures. This paper presents the first large-scale empirical study characterizing physics failures in PE-based software. We investigate three research questions addressing the manifestations of physics failures, the effectiveness of detection techniques, and developer perceptions of current detection practices. Our contributions include: (1) a taxonomy of physics failure manifestations; (2) a comprehensive evaluation of detection methods including deep learning, prompt-based techniques, and large multimodal models; and (3) actionable insights from developer experiences for improving detection approaches. To support future research, we release PhysiXFails, code, and other materials at https://sites.google.com/view/physics-failure-detection.