# Learning Physical Intuition of Block Towers by Example

arXiv: https://arxiv.org/abs/1603.01312 · HTML: https://mlsanity.com/paper/1603.01312/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Mar 2016.

## Built on by (most cited first, 30 of 68)

- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Physion: Evaluating Physical Prediction from Vision in Humans and Machines](https://mlsanity.com/paper/2106.08261/) (2021)
- [Mind's Eye: Grounded Language Model Reasoning through Simulation](https://mlsanity.com/paper/2210.05359/) (2022)
- [Learning to Compose Visual Relations](https://mlsanity.com/paper/2111.09297/) (2021)
- [Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapes](https://mlsanity.com/paper/2110.06192/) (2021)
- [Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning](https://mlsanity.com/paper/2103.16564/) (2021)
- [ComPhy: Compositional Physical Reasoning of Objects and Events from Videos](https://mlsanity.com/paper/2205.01089/) (2022)
- [Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language](https://mlsanity.com/paper/2110.15358/) (2021)
- [Learning Physical Dynamics with Subequivariant Graph Neural Networks](https://mlsanity.com/paper/2210.06876/) (2022)
- [Visual cognition in multimodal large language models](https://mlsanity.com/paper/2311.16093/) (2023)
- [From Machine Learning to Robotics: Challenges and Opportunities for Embodied Intelligence](https://mlsanity.com/paper/2110.15245/) (2021)
- [What is the Visual Cognition Gap between Humans and Multimodal LLMs?](https://mlsanity.com/paper/2406.10424/) (2024)
- [GRASP: A novel benchmark for evaluating language GRounding And Situated Physics understanding in multimodal language models](https://mlsanity.com/paper/2311.09048/) (2023)
- [SwingBot: Learning Physical Features from In-hand Tactile Exploration for Dynamic Swing-up Manipulation](https://mlsanity.com/paper/2101.11812/) (2021)
- [CausalCity: Complex Simulations with Agency for Causal Discovery and Reasoning](https://mlsanity.com/paper/2106.13364/) (2021)
- [PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videos](https://mlsanity.com/paper/2412.01800/) (2024)
- [A Survey on Machine Learning Approaches for Modelling Intuitive Physics](https://mlsanity.com/paper/2202.06481/) (2022)
- [I-PHYRE: Interactive Physical Reasoning](https://mlsanity.com/paper/2312.03009/) (2023)
- [Filtered-CoPhy: Unsupervised Learning of Counterfactual Physics in Pixel Space](https://mlsanity.com/paper/2202.00368/) (2022)
- [Estimating counterfactual treatment outcomes over time in complex multiagent scenarios](https://mlsanity.com/paper/2206.01900/) (2022)
- [3D-IntPhys: Towards More Generalized 3D-grounded Visual Intuitive Physics under Challenging Scenes](https://mlsanity.com/paper/2304.11470/) (2023)
- [Compositional Physical Reasoning of Objects and Events from Videos](https://mlsanity.com/paper/2408.02687/) (2024)
- [InterDyn: Controllable Interactive Dynamics with Video Diffusion Models](https://mlsanity.com/paper/2412.11785/) (2024)
- [OIL: Observational Imitation Learning](https://mlsanity.com/paper/1803.01129/) (2018)
- [Active Learning of Abstract Plan Feasibility](https://mlsanity.com/paper/2107.00683/) (2021)
- [On the Learning Mechanisms in Physical Reasoning](https://mlsanity.com/paper/2210.02075/) (2022)
- [Visual Jenga: Discovering Object Dependencies via Counterfactual Inpainting](https://mlsanity.com/paper/2503.21770/) (2025)
- [Physics Context Builders: A Modular Framework for Physical Reasoning in Vision-Language Models](https://mlsanity.com/paper/2412.08619/) (2024)
- [PIP: Physical Interaction Prediction via Mental Simulation with Span Selection](https://mlsanity.com/paper/2109.04683/) (2021)
- [A Benchmark for Modeling Violation-of-Expectation in Physical Reasoning Across Event Categories](https://mlsanity.com/paper/2111.08826/) (2021)

## Abstract (arXiv)

Wooden blocks are a common toy for infants, allowing them to develop motor skills and gain intuition about the physical behavior of the world. In this paper, we explore the ability of deep feed-forward models to learn such intuitive physics. Using a 3D game engine, we create small towers of wooden blocks whose stability is randomized and render them collapsing (or remaining upright). This data allows us to train large convolutional network models which can accurately predict the outcome, as well as estimating the block trajectories. The models are also able to generalize in two important ways: (i) to new physical scenarios, e.g. towers with an additional block and (ii) to images of real wooden blocks, where it obtains a performance comparable to human subjects.