# Inferring the Future by Imagining the Past

arXiv: https://arxiv.org/abs/2305.17195 · HTML: https://mlsanity.com/paper/2305.17195/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 65 references that are in the map.
- First posted on arXiv 26 May 2023.

## Builds on (most cited first, 9 of 9)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress](https://mlsanity.com/paper/1806.06877/) (2018)
- [Online Bayesian Goal Inference for Boundedly-Rational Planning Agents](https://mlsanity.com/paper/2006.07532/) (2020)
- [Preferences Implicit in the State of the World](https://mlsanity.com/paper/1902.04198/) (2019)
- [PHASE: PHysically-grounded Abstract Social Events for Machine Social Perception](https://mlsanity.com/paper/2103.01933/) (2021)
- [Designing Perceptual Puzzles by Differentiating Probabilistic Programs](https://mlsanity.com/paper/2204.12301/) (2022)
- [Physical problem solving: Joint planning with symbolic, geometric, and dynamic constraints](https://mlsanity.com/paper/1707.08212/) (2017)
- [Learning What To Do by Simulating the Past](https://mlsanity.com/paper/2104.03946/) (2021)
- [Explaining intuitive difficulty judgments by modeling physical effort and risk](https://mlsanity.com/paper/1905.04445/) (2019)

## Built on by (most cited first, 4 of 4)

- [Building Machines that Learn and Think with People](https://mlsanity.com/paper/2408.03943/) (2024)
- [Infinite Ends from Finite Samples: Open-Ended Goal Inference as Top-Down Bayesian Filtering of Bottom-Up Proposals](https://mlsanity.com/paper/2407.16770/) (2024)
- [WatChat: Explaining perplexing programs by debugging mental models](https://mlsanity.com/paper/2403.05334/) (2024)
- [Reading Cognition as Decisions Unfold in Words: A Factorized Inverse Decision Model](https://mlsanity.com/paper/2608.09222/) (2026)

## Abstract (arXiv)

A single panel of a comic book can say a lot: it can depict not only where the characters currently are, but also their motions, their motivations, their emotions, and what they might do next. More generally, humans routinely infer complex sequences of past and future events from a *static snapshot* of a *dynamic scene*, even in situations they have never seen before. In this paper, we model how humans make such rapid and flexible inferences. Building on a long line of work in cognitive science, we offer a Monte Carlo algorithm whose inferences correlate well with human intuitions in a wide variety of domains, while only using a small, cognitively-plausible number of samples. Our key technical insight is a surprising connection between our inference problem and Monte Carlo path tracing, which allows us to apply decades of ideas from the computer graphics community to this seemingly-unrelated theory of mind task.