# Robot Motion Planning as Video Prediction: A Spatio-Temporal Neural Network-based Motion Planner

arXiv: https://arxiv.org/abs/2208.11287 · HTML: https://mlsanity.com/paper/2208.11287/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 28 references that are in the map.
- First posted on arXiv 24 Aug 2022.

## Builds on (most cited first, 16 of 16)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Learning Sampling Distributions for Robot Motion Planning](https://mlsanity.com/paper/1709.05448/) (2017)
- [Motion Planning Networks](https://mlsanity.com/paper/1806.05767/) (2018)
- [Motion Planning Networks: Bridging the Gap Between Learning-based and Classical Motion Planners](https://mlsanity.com/paper/1907.06013/) (2019)
- [Informed RRT*: Optimal Sampling-based Path Planning Focused via Direct Sampling of an Admissible Ellipsoidal Heuristic](https://mlsanity.com/paper/1404.2334/) (2014)
- [Robot Motion Planning in Learned Latent Spaces](https://mlsanity.com/paper/1807.10366/) (2018)
- [Batch Informed Trees (BIT*): Informed Asymptotically Optimal Anytime Search](https://mlsanity.com/paper/1707.01888/) (2017)
- [Neural Path Planning: Fixed Time, Near-Optimal Path Generation via Oracle Imitation](https://mlsanity.com/paper/1904.11102/) (2019)
- [Deeply Informed Neural Sampling for Robot Motion Planning](https://mlsanity.com/paper/1809.10252/) (2018)
- [LEGO: Leveraging Experience in Roadmap Generation for Sampling-Based Planning](https://mlsanity.com/paper/1907.09574/) (2019)
- [Learning to Plan in High Dimensions via Neural Exploration-Exploitation Trees](https://mlsanity.com/paper/1903.00070/) (2019)
- [Graph Neural Networks for Motion Planning](https://mlsanity.com/paper/2006.06248/) (2020)
- [Learning Latent State Spaces for Planning through Reward Prediction](https://mlsanity.com/paper/1912.04201/) (2019)
- [Waypoint Planning Networks](https://mlsanity.com/paper/2105.00312/) (2021)
- [Learning and Using Abstractions for Robot Planning](https://mlsanity.com/paper/2012.00658/) (2020)

## Built on by (most cited first, 2 of 2)

- [Learning Physical Dynamics for Object-centric Visual Prediction](https://mlsanity.com/paper/2403.10079/) (2024)
- [Keypoint-based Diffusion for Robotic Motion Planning on the NICOL Robot](https://mlsanity.com/paper/2509.04076/) (2025)

## Abstract (arXiv)

Neural network (NN)-based methods have emerged as an attractive approach for robot motion planning due to strong learning capabilities of NN models and their inherently high parallelism. Despite the current development in this direction, the efficient capture and processing of important sequential and spatial information, in a direct and simultaneous way, is still relatively under-explored. To overcome the challenge and unlock the potentials of neural networks for motion planning tasks, in this paper, we propose STP-Net, an end-to-end learning framework that can fully extract and leverage important spatio-temporal information to form an efficient neural motion planner. By interpreting the movement of the robot as a video clip, robot motion planning is transformed to a video prediction task that can be performed by STP-Net in both spatially and temporally efficient ways. Empirical evaluations across different seen and unseen environments show that, with nearly 100% accuracy (aka, success rate), STP-Net demonstrates very promising performance with respect to both planning speed and path cost. Compared with existing NN-based motion planners, STP-Net achieves at least 5x, 2.6x and 1.8x faster speed with lower path cost on 2D Random Forest, 2D Maze and 3D Random Forest environments, respectively. Furthermore, STP-Net can quickly and simultaneously compute multiple near-optimal paths in multi-robot motion planning tasks