# X-VoE: Measuring eXplanatory Violation of Expectation in Physical Events

arXiv: https://arxiv.org/abs/2308.10441 · HTML: https://mlsanity.com/paper/2308.10441/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 46 references that are in the map.
- First posted on arXiv 21 Aug 2023.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [MONet: Unsupervised Scene Decomposition and Representation](https://mlsanity.com/paper/1901.11390/) (2019)
- [Deep multi-scale video prediction beyond mean square error](https://mlsanity.com/paper/1511.05440/) (2015)
- [Unsupervised Learning for Physical Interaction through Video Prediction](https://mlsanity.com/paper/1605.07157/) (2016)
- [Stochastic Variational Video Prediction](https://mlsanity.com/paper/1710.11252/) (2017)
- [Video Frame Synthesis using Deep Voxel Flow](https://mlsanity.com/paper/1702.02463/) (2017)
- [Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning](https://mlsanity.com/paper/1605.08104/) (2016)
- [Disentangling Physical Dynamics from Unknown Factors for Unsupervised Video Prediction](https://mlsanity.com/paper/2003.01460/) (2020)
- [Scaling Autoregressive Video Models](https://mlsanity.com/paper/1906.02634/) (2019)
- [Decomposing Motion and Content for Natural Video Sequence Prediction](https://mlsanity.com/paper/1706.08033/) (2017)
- [A Good Image Generator Is What You Need for High-Resolution Video Synthesis](https://mlsanity.com/paper/2104.15069/) (2021)
- [Unsupervised Learning of Disentangled Representations from Video](https://mlsanity.com/paper/1705.10915/) (2017)
- [Learning to Decompose and Disentangle Representations for Video Prediction](https://mlsanity.com/paper/1806.04166/) (2018)
- [A Compositional Object-Based Approach to Learning Physical Dynamics](https://mlsanity.com/paper/1612.00341/) (2016)
- [Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs](https://mlsanity.com/paper/1901.07017/) (2019)
- [Visual Interaction Networks](https://mlsanity.com/paper/1706.01433/) (2017)
- [IntPhys: A Framework and Benchmark for Visual Intuitive Physics Reasoning](https://mlsanity.com/paper/1803.07616/) (2018)
- [Learning Physical Intuition of Block Towers by Example](https://mlsanity.com/paper/1603.01312/) (2016)
- [Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning](https://mlsanity.com/paper/2103.16564/) (2021)
- [Attention over learned object embeddings enables complex visual reasoning](https://mlsanity.com/paper/2012.08508/) (2020)
- [Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language](https://mlsanity.com/paper/2110.15358/) (2021)
- [SDCNet: Video Prediction Using Spatially-Displaced Convolution](https://mlsanity.com/paper/1811.00684/) (2018)
- [The Emergence of Objectness: Learning Zero-Shot Segmentation from Videos](https://mlsanity.com/paper/2111.06394/) (2021)
- [Probing Physics Knowledge Using Tools from Developmental Psychology](https://mlsanity.com/paper/1804.01128/) (2018)
- [Occlusion resistant learning of intuitive physics from videos](https://mlsanity.com/paper/2005.00069/) (2020)
- [A Benchmark for Modeling Violation-of-Expectation in Physical Reasoning Across Event Categories](https://mlsanity.com/paper/2111.08826/) (2021)

## Built on by (most cited first, 3 of 3)

- [Learning Physics-Grounded 4D Dynamics with Neural Gaussian Force Fields](https://mlsanity.com/paper/2602.00148/) (2026)
- [Neural Force Field: Few-shot Learning of Generalized Physical Reasoning](https://mlsanity.com/paper/2502.08987/) (2025)
- [FADE: From Passive Verification to Active Discovery in Counterfactual Video Understanding](https://mlsanity.com/paper/2608.10764/) (2026)

## Abstract (arXiv)

Intuitive physics is pivotal for human understanding of the physical world, enabling prediction and interpretation of events even in infancy. Nonetheless, replicating this level of intuitive physics in artificial intelligence (AI) remains a formidable challenge. This study introduces X-VoE, a comprehensive benchmark dataset, to assess AI agents' grasp of intuitive physics. Built on the developmental psychology-rooted Violation of Expectation (VoE) paradigm, X-VoE establishes a higher bar for the explanatory capacities of intuitive physics models. Each VoE scenario within X-VoE encompasses three distinct settings, probing models' comprehension of events and their underlying explanations. Beyond model evaluation, we present an explanation-based learning system that captures physics dynamics and infers occluded object states solely from visual sequences, without explicit occlusion labels. Experimental outcomes highlight our model's alignment with human commonsense when tested against X-VoE. A remarkable feature is our model's ability to visually expound VoE events by reconstructing concealed scenes. Concluding, we discuss the findings' implications and outline future research directions. Through X-VoE, we catalyze the advancement of AI endowed with human-like intuitive physics capabilities.