# Answering Visual What-If Questions: From Actions to Predicted Scene Descriptions

arXiv: https://arxiv.org/abs/1809.03707 · HTML: https://mlsanity.com/paper/1809.03707/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Sep 2018.

## Built on by (most cited first, 10 of 10)

- [CLEVR_HYP: A Challenge Dataset and Baselines for Visual Question Answering with Hypothetical Actions over Images](https://mlsanity.com/paper/2104.05981/) (2021)
- [What Makes a Maze Look Like a Maze?](https://mlsanity.com/paper/2409.08202/) (2024)
- [Reasoning about Actions over Visual and Linguistic Modalities: A Survey](https://mlsanity.com/paper/2207.07568/) (2022)
- [A Benchmark for Modeling Violation-of-Expectation in Physical Reasoning Across Event Categories](https://mlsanity.com/paper/2111.08826/) (2021)
- [CoSIm: Commonsense Reasoning for Counterfactual Scene Imagination](https://mlsanity.com/paper/2207.03961/) (2022)
- [AVoE: A Synthetic 3D Dataset on Understanding Violation of Expectation for Artificial Cognition](https://mlsanity.com/paper/2110.05836/) (2021)
- [Learning Action-Effect Dynamics for Hypothetical Vision-Language Reasoning Task](https://mlsanity.com/paper/2212.03866/) (2022)
- [Understanding in Artificial Intelligence](https://mlsanity.com/paper/2101.06573/) (2021)
- [Combining Commonsense Reasoning and Knowledge Acquisition to Guide Deep Learning in Robotics](https://mlsanity.com/paper/2201.10266/) (2022)
- [Think before You Simulate: Symbolic Reasoning to Orchestrate Neural Computation for Counterfactual Question Answering](https://mlsanity.com/paper/2506.10753/) (2025)

## Abstract (arXiv)

In-depth scene descriptions and question answering tasks have greatly increased the scope of today's definition of scene understanding. While such tasks are in principle open ended, current formulations primarily focus on describing only the current state of the scenes under consideration. In contrast, in this paper, we focus on the future states of the scenes which are also conditioned on actions. We posit this as a question answering task, where an answer has to be given about a future scene state, given observations of the current scene, and a question that includes a hypothetical action. Our solution is a hybrid model which integrates a physics engine into a question answering architecture in order to anticipate future scene states resulting from object-object interactions caused by an action. We demonstrate first results on this challenging new problem and compare to baselines, where we outperform fully data-driven end-to-end learning approaches.