# Robot Motion Planning in Learned Latent Spaces

arXiv: https://arxiv.org/abs/1807.10366 · HTML: https://mlsanity.com/paper/1807.10366/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jul 2018.

## Built on by (most cited first, 30 of 42)

- [Video as the New Language for Real-World Decision Making](https://mlsanity.com/paper/2402.17139/) (2024)
- [Learning Multi-Object Dynamics with Compositional Neural Radiance Fields](https://mlsanity.com/paper/2202.11855/) (2022)
- [Generalizing Safety Beyond Collision-Avoidance via Latent-Space Reachability Analysis](https://mlsanity.com/paper/2502.00935/) (2025)
- [Potential Based Diffusion Motion Planning](https://mlsanity.com/paper/2407.06169/) (2024)
- [Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning](https://mlsanity.com/paper/2111.03189/) (2021)
- [NTFields: Neural Time Fields for Physics-Informed Robot Motion Planning](https://mlsanity.com/paper/2210.00120/) (2022)
- [Reducing Collision Checking for Sampling-Based Motion Planning Using Graph Neural Networks](https://mlsanity.com/paper/2210.08864/) (2022)
- [Reinforcement Learning for Collision-free Flight Exploiting Deep Collision Encoding](https://mlsanity.com/paper/2402.03947/) (2024)
- [MAVRL: Learn to Fly in Cluttered Environments with Varying Speed](https://mlsanity.com/paper/2402.08381/) (2024)
- [It Takes Two: Learning to Plan for Human-Robot Cooperative Carrying](https://mlsanity.com/paper/2209.12890/) (2022)
- [LS3: Latent Space Safe Sets for Long-Horizon Visuomotor Control of Sparse Reward Iterative Tasks](https://mlsanity.com/paper/2107.04775/) (2021)
- [MPC-MPNet: Model-Predictive Motion Planning Networks for Fast, Near-Optimal Planning under Kinodynamic Constraints](https://mlsanity.com/paper/2101.06798/) (2021)
- [Reaching Through Latent Space: From Joint Statistics to Path Planning in Manipulation](https://mlsanity.com/paper/2210.11779/) (2022)
- [Learning Sampling Dictionaries for Efficient and Generalizable Robot Motion Planning with Transformers](https://mlsanity.com/paper/2306.00851/) (2023)
- [A Survey on the Integration of Machine Learning with Sampling-based Motion Planning](https://mlsanity.com/paper/2211.08368/) (2022)
- [Physics-informed Temporal Difference Metric Learning for Robot Motion Planning](https://mlsanity.com/paper/2505.05691/) (2025)
- [Contrastive Representations for Temporal Reasoning](https://mlsanity.com/paper/2508.13113/) (2025)
- [Planning from Pixels in Environments with Combinatorially Hard Search Spaces](https://mlsanity.com/paper/2110.06149/) (2021)
- [Learning-based Motion Planning in Dynamic Environments Using GNNs and Temporal Encoding](https://mlsanity.com/paper/2210.08408/) (2022)
- [Enabling Visual Action Planning for Object Manipulation through Latent Space Roadmap](https://mlsanity.com/paper/2103.02554/) (2021)
- [Learning Sequential Latent Variable Models from Multimodal Time Series Data](https://mlsanity.com/paper/2204.10419/) (2022)
- [Leveraging Scene Embeddings for Gradient-Based Motion Planning in Latent Space](https://mlsanity.com/paper/2303.03364/) (2023)
- [Approximating Constraint Manifolds Using Generative Models for Sampling-Based Constrained Motion Planning](https://mlsanity.com/paper/2204.06791/) (2022)
- [Robot Motion Planning as Video Prediction: A Spatio-Temporal Neural Network-based Motion Planner](https://mlsanity.com/paper/2208.11287/) (2022)
- [Unified Control Framework for Real-Time Interception and Obstacle Avoidance of Fast-Moving Objects with Diffusion Variational Autoencoder](https://mlsanity.com/paper/2209.13628/) (2022)
- [CLAMGen: Closed-Loop Arm Motion Generation via Multi-view Vision-Based RL](https://mlsanity.com/paper/2103.13267/) (2021)
- [Comparing Reconstruction- and Contrastive-based Models for Visual Task Planning](https://mlsanity.com/paper/2109.06737/) (2021)
- [SERA: Safe and Efficient Reactive Obstacle Avoidance for Collaborative Robotic Planning in Unstructured Environments](https://mlsanity.com/paper/2203.13821/) (2022)
- [A Framework for Guided Motion Planning](https://mlsanity.com/paper/2404.03133/) (2024)
- [Transformer-Enhanced Motion Planner: Attention-Guided Sampling for State-Specific Decision Making](https://mlsanity.com/paper/2404.19403/) (2024)

## Abstract (arXiv)

This paper presents Latent Sampling-based Motion Planning (L-SBMP), a methodology towards computing motion plans for complex robotic systems by learning a plannable latent representation. Recent works in control of robotic systems have effectively leveraged local, low-dimensional embeddings of high-dimensional dynamics. In this paper we combine these recent advances with techniques from sampling-based motion planning (SBMP) in order to design a methodology capable of planning for high-dimensional robotic systems beyond the reach of traditional approaches (e.g., humanoids, or even systems where planning occurs in the visual space). Specifically, the learned latent space is constructed through an autoencoding network, a dynamics network, and a collision checking network, which mirror the three main algorithmic primitives of SBMP, namely state sampling, local steering, and collision checking. Notably, these networks can be trained through only raw data of the system's states and actions along with a supervising collision checker. Building upon these networks, an RRT-based algorithm is used to plan motions directly in the latent space - we refer to this exploration algorithm as Learned Latent RRT (L2RRT). This algorithm globally explores the latent space and is capable of generalizing to new environments. The overall methodology is demonstrated on two planning problems, namely a visual planning problem, whereby planning happens in the visual (pixel) space, and a humanoid robot planning problem.