# SPACE: A Simulator for Physical Interactions and Causal Learning in 3D Environments

arXiv: https://arxiv.org/abs/2108.06180 · HTML: https://mlsanity.com/paper/2108.06180/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 37 references that are in the map.
- First posted on arXiv 13 Aug 2021.

## Builds on (most cited first, 14 of 14)

- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [CLEVRER: CoLlision Events for Video REpresentation and Reasoning](https://mlsanity.com/paper/1910.01442/) (2019)
- [Future Frame Prediction for Anomaly Detection -- A New Baseline](https://mlsanity.com/paper/1712.09867/) (2017)
- [A Survey of Embodied AI: From Simulators to Research Tasks](https://mlsanity.com/paper/2103.04918/) (2021)
- [Human Action Recognition and Prediction: A Survey](https://mlsanity.com/paper/1806.11230/) (2018)
- [Disentangling Physical Dynamics from Unknown Factors for Unsupervised Video Prediction](https://mlsanity.com/paper/2003.01460/) (2020)
- [Long-term Human Motion Prediction with Scene Context](https://mlsanity.com/paper/2007.03672/) (2020)
- [CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning](https://mlsanity.com/paper/1910.04744/) (2019)
- [Peeking into the Future: Predicting Future Person Activities and Locations in Videos](https://mlsanity.com/paper/1902.03748/) (2019)
- [CoPhy: Counterfactual Learning of Physical Dynamics](https://mlsanity.com/paper/1909.12000/) (2019)
- [AGENT: A Benchmark for Core Psychological Reasoning](https://mlsanity.com/paper/2102.12321/) (2021)
- [PHASE: PHysically-grounded Abstract Social Events for Machine Social Perception](https://mlsanity.com/paper/2103.01933/) (2021)
- [Actionet: An Interactive End-To-End Platform For Task-Based Data Collection And Augmentation In 3D Environment](https://mlsanity.com/paper/2010.01357/) (2020)

## Built on by (most cited first, 9 of 9)

- [Unfolding Spatial Cognition: Evaluating Multimodal Models on Visual Simulations](https://mlsanity.com/paper/2506.04633/) (2025)
- [VLS: Steering Pretrained Robot Policies via Vision-Language Models](https://mlsanity.com/paper/2602.03973/) (2026)
- [MVTrans: Multi-View Perception of Transparent Objects](https://mlsanity.com/paper/2302.11683/) (2023)
- [Towards Visual Discrimination and Reasoning of Real-World Physical Dynamics: Physics-Grounded Anomaly Detection](https://mlsanity.com/paper/2503.03562/) (2025)
- [PIP: Physical Interaction Prediction via Mental Simulation with Span Selection](https://mlsanity.com/paper/2109.04683/) (2021)
- [A Benchmark for Modeling Violation-of-Expectation in Physical Reasoning Across Event Categories](https://mlsanity.com/paper/2111.08826/) (2021)
- [Benchmarks for Physical Reasoning AI](https://mlsanity.com/paper/2312.10728/) (2023)
- [AVoE: A Synthetic 3D Dataset on Understanding Violation of Expectation for Artificial Cognition](https://mlsanity.com/paper/2110.05836/) (2021)
- [Take A Step Back: Rethinking the Two Stages in Visual Reasoning](https://mlsanity.com/paper/2407.19666/) (2024)

## Abstract (arXiv)

Recent advancements in deep learning, computer vision, and embodied AI have given rise to synthetic causal reasoning video datasets. These datasets facilitate the development of AI algorithms that can reason about physical interactions between objects. However, datasets thus far have primarily focused on elementary physical events such as rolling or falling. There is currently a scarcity of datasets that focus on the physical interactions that humans perform daily with objects in the real world. To address this scarcity, we introduce SPACE: A Simulator for Physical Interactions and Causal Learning in 3D Environments. The SPACE simulator allows us to generate the SPACE dataset, a synthetic video dataset in a 3D environment, to systematically evaluate physics-based models on a range of physical causal reasoning tasks. Inspired by daily object interactions, the SPACE dataset comprises videos depicting three types of physical events: containment, stability and contact. These events make up the vast majority of the basic physical interactions between objects. We then further evaluate it with a state-of-the-art physics-based deep model and show that the SPACE dataset improves the learning of intuitive physics with an approach inspired by curriculum learning. Repository: https://github.com/jiafei1224/SPACE