# Prospection: Interpretable Plans From Language By Predicting the Future

arXiv: https://arxiv.org/abs/1903.08309 · HTML: https://mlsanity.com/paper/1903.08309/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Mar 2019.

## Built on by (most cited first, 21 of 21)

- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [CLIPort: What and Where Pathways for Robotic Manipulation](https://mlsanity.com/paper/2109.12098/) (2021)
- [Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation](https://mlsanity.com/paper/2209.05451/) (2022)
- [A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution](https://mlsanity.com/paper/2107.05612/) (2021)
- [Correcting Robot Plans with Natural Language Feedback](https://mlsanity.com/paper/2204.05186/) (2022)
- [StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objects](https://mlsanity.com/paper/2110.10189/) (2021)
- [SORNet: Spatial Object-Centric Representations for Sequential Manipulation](https://mlsanity.com/paper/2109.03891/) (2021)
- [Distilling Script Knowledge from Large Language Models for Constrained Language Planning](https://mlsanity.com/paper/2305.05252/) (2023)
- [Language Grounding with 3D Objects](https://mlsanity.com/paper/2107.12514/) (2021)
- [Latent Execution for Neural Program Synthesis](https://mlsanity.com/paper/2107.00101/) (2021)
- [LISA: Learning Interpretable Skill Abstractions from Language](https://mlsanity.com/paper/2203.00054/) (2022)
- [Vision-Language Interpreter for Robot Task Planning](https://mlsanity.com/paper/2311.00967/) (2023)
- [Trajectory Prediction with Linguistic Representations](https://mlsanity.com/paper/2110.09741/) (2021)
- [Learning Neuro-symbolic Programs for Language Guided Robot Manipulation](https://mlsanity.com/paper/2211.06652/) (2022)
- [Policy Adaptation from Foundation Model Feedback](https://mlsanity.com/paper/2212.07398/) (2022)
- [Robi Butler: Multimodal Remote Interaction with a Household Robot Assistant](https://mlsanity.com/paper/2409.20548/) (2024)
- [Spatial-Language Attention Policies for Efficient Robot Learning](https://mlsanity.com/paper/2304.11235/) (2023)
- [DASH: Modularized Human Manipulation Simulation with Vision and Language for Embodied AI](https://mlsanity.com/paper/2108.12536/) (2021)
- [tagE: Enabling an Embodied Agent to Understand Human Instructions](https://mlsanity.com/paper/2310.15605/) (2023)
- [Automating Complex Document Workflows via Stepwise and Rollback-Enabled Operation Orchestration](https://mlsanity.com/paper/2512.04445/) (2025)
- [AI-IoT-Robotics Integration: Survey of Frameworks, Emerging Trends, and the Path Toward Connected Robotics](https://mlsanity.com/paper/2606.01015/) (2026)

## Abstract (arXiv)

High-level human instructions often correspond to behaviors with multiple implicit steps. In order for robots to be useful in the real world, they must be able to to reason over both motions and intermediate goals implied by human instructions. In this work, we propose a framework for learning representations that convert from a natural-language command to a sequence of intermediate goals for execution on a robot. A key feature of this framework is prospection, training an agent not just to correctly execute the prescribed command, but to predict a horizon of consequences of an action before taking it. We demonstrate the fidelity of plans generated by our framework when interpreting real, crowd-sourced natural language commands for a robot in simulated scenes.