# Latent Space Roadmap for Visual Action Planning of Deformable and Rigid Object Manipulation

arXiv: https://arxiv.org/abs/2003.08974 · HTML: https://mlsanity.com/paper/2003.08974/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Mar 2020.

## Built on by (most cited first, 25 of 25)

- [FabricFlowNet: Bimanual Cloth Manipulation with a Flow-based Policy](https://mlsanity.com/paper/2111.05623/) (2021)
- [ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulation](https://mlsanity.com/paper/2203.06856/) (2022)
- [VisuoSpatial Foresight for Physical Sequential Fabric Manipulation](https://mlsanity.com/paper/2102.09754/) (2021)
- [LS3: Latent Space Safe Sets for Long-Horizon Visuomotor Control of Sparse Reward Iterative Tasks](https://mlsanity.com/paper/2107.04775/) (2021)
- [EDO-Net: Learning Elastic Properties of Deformable Objects from Graph Dynamics](https://mlsanity.com/paper/2209.08996/) (2022)
- [GeomCA: Geometric Evaluation of Data Representations](https://mlsanity.com/paper/2105.12486/) (2021)
- [ActionDiffusion: An Action-aware Diffusion Model for Procedure Planning in Instructional Videos](https://mlsanity.com/paper/2403.08591/) (2024)
- [OSCAR: Data-Driven Operational Space Control for Adaptive and Robust Robot Manipulation](https://mlsanity.com/paper/2110.00704/) (2021)
- [Delaunay Component Analysis for Evaluation of Data Representations](https://mlsanity.com/paper/2202.06866/) (2022)
- [Neural Constraint Satisfaction: Hierarchical Abstraction for Combinatorial Generalization in Object Rearrangement](https://mlsanity.com/paper/2303.11373/) (2023)
- [General-purpose Clothes Manipulation with Semantic Keypoints](https://mlsanity.com/paper/2408.08160/) (2024)
- [Enabling Visual Action Planning for Object Manipulation through Latent Space Roadmap](https://mlsanity.com/paper/2103.02554/) (2021)
- [Learning Sequential Latent Variable Models from Multimodal Time Series Data](https://mlsanity.com/paper/2204.10419/) (2022)
- [Semantic State Estimation in Cloth Manipulation Tasks](https://mlsanity.com/paper/2203.11647/) (2022)
- [Robustness without Wrinkles: Parallel Simulation and Robust MPC for Certified Deformable Manipulation](https://mlsanity.com/paper/2606.14188/) (2026)
- [Implicit Neural-Representation Learning for Elastic Deformable-Object Manipulations](https://mlsanity.com/paper/2505.00500/) (2025)
- [Instant-Fold: In-Context Imitation Learning for Deformable Object Manipulation](https://mlsanity.com/paper/2606.04269/) (2026)
- [Comparing Reconstruction- and Contrastive-based Models for Visual Task Planning](https://mlsanity.com/paper/2109.06737/) (2021)
- [Training and Evaluation of Deep Policies using Reinforcement Learning and Generative Models](https://mlsanity.com/paper/2204.08573/) (2022)
- [Learning a General Model: Folding Clothing with Topological Dynamics](https://mlsanity.com/paper/2504.20720/) (2025)
- [Understanding Physical Properties of Unseen Deformable Objects by Leveraging Large Language Models and Robot Actions](https://mlsanity.com/paper/2506.03760/) (2025)
- [CLASP: General-Purpose Clothes Manipulation with Semantic Keypoints](https://mlsanity.com/paper/2507.19983/) (2025)
- [The dGLI Cloth Coordinates: A Topological Representation for Semantic Classification of Cloth States](https://mlsanity.com/paper/2209.09191/) (2022)
- [V-MORALS: Visual Morse Graph-Aided Estimation of Regions of Attraction in a Learned Latent Space](https://mlsanity.com/paper/2602.23524/) (2026)
- [V2P-Manip: Learning Dexterous Manipulation from Monocular Human Videos](https://mlsanity.com/paper/2606.16436/) (2026)

## Abstract (arXiv)

We present a framework for visual action planning of complex manipulation tasks with high-dimensional state spaces such as manipulation of deformable objects. Planning is performed in a low-dimensional latent state space that embeds images. We define and implement a Latent Space Roadmap (LSR) which is a graph-based structure that globally captures the latent system dynamics. Our framework consists of two main components: a Visual Foresight Module (VFM) that generates a visual plan as a sequence of images, and an Action Proposal Network (APN) that predicts the actions between them. We show the effectiveness of the method on a simulated box stacking task as well as a T-shirt folding task performed with a real robot.