mlsanityData to 7 Oct 2026

Enabling Visual Action Planning for Object Manipulation through Latent Space Roadmap

arXiv 2103.02554 ↗First posted 3 Mar 2021cs.RO cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

We present a framework for visual action planning of complex manipulation tasks with high-dimensional state spaces, focusing on manipulation of deformable objects. We propose a Latent Space Roadmap (LSR) for task planning which is a graph-based structure globally capturing the system dynamics in a low-dimensional latent space. Our framework consists of three parts: (1) a Mapping Module (MM) that maps observations given in the form of images into a structured latent space extracting the respective states as well as generates observations from the latent states, (2) the LSR which builds and connects clusters containing similar states in order to find the latent plans between start and goal states extracted by MM, and (3) the Action Proposal Module that complements the latent plan found by the LSR with the corresponding actions. We present a thorough investigation of our framework on simulated box stacking and rope/box manipulation tasks, and a folding task executed on a real robot.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 35 of its 83 references that are in the map. First posted on arXiv 3 Mar 2021.

Builds on

35 of 83 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2017Neural Discrete Representation Learningcited by 4,650
  3. 2013Auto-Encoding Variational Bayescited by 3,980
  4. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  5. 2018Understanding disentangling in β-VAEcited by 332
  6. 2019Learning Latent Plans from Playcited by 310
  7. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  8. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 266
  9. 2019Contrastive Learning of Structured World Modelscited by 206
  10. 2018Semi-parametric Topological Memory for Navigationcited by 203
  11. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 188
  12. 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 171
23 more
  1. 2018Sim-to-Real Reinforcement Learning for Deformable Object Manipulationcited by 124
  2. 2020SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulationcited by 118
  3. 2017Combining Self-Supervised Learning and Imitation for Vision-Based Rope Manipulationcited by 90
  4. 2018Understanding and Improving Interpolation in Autoencoders via an Adversarial Regularizercited by 87
  5. 2020Learning Predictive Representations for Deformable Objects Using Contrastive Estimationcited by 72
  6. 2019Motion Planning Networks: Bridging the Gap Between Learning-based and Classical Motion Plannerscited by 62
  7. 2020Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictorscited by 60
  8. 2018Learning Plannable Representations with Causal InfoGANcited by 56
  9. 2019Towards Practical Multi-Object Manipulation using Relational Reinforcement Learningcited by 54
  10. 2019Deep Imitation Learning of Sequential Fabric Smoothing From an Algorithmic Supervisorcited by 44
  11. 2018Robot Motion Planning in Learned Latent Spacescited by 40
  12. 2020VisuoSpatial Foresight for Multi-Step, Multi-Task Fabric Manipulationcited by 40
  13. 2019Learning Robotic Manipulation through Visual Planning and Actingcited by 39
  14. 2020Sparse Graphical Memory for Robust Planningcited by 30
  15. 2018Universal Planning Networkscited by 29
  16. 2020Broadly-Exploring, Local-Policy Trees for Long-Horizon Task Planningcited by 26
  17. 2020Hallucinative Topological Memory for Zero-Shot Visual Planningcited by 25
  18. 2020Latent Space Roadmap for Visual Action Planning of Deformable and Rigid Object Manipulationcited by 25
  19. 2020Learning Arbitrary-Goal Fabric Folding with One Hour of Real Robot Experiencecited by 24
  20. 2020Cloth Region Segmentation for Robust Grasp Selectioncited by 15
  21. 2018Cloth Manipulation Using Random-Forest-Based Imitation Learningcited by 7
  22. 2020Autoencoder Image Interpolation by Shaping the Latent Spacecited by 2
  23. 2020LaSeSOM: A Latent and Semantic Representation Framework for Soft Object Manipulationcited by 1

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 1; 2023: 1; 2024: 0; 2025: 1.

4 papers.

  1. 2023Transfer Learning in Robotics: An Upcoming Breakthrough? A Review of Promises and Challengescited by 17
  2. 2022EDO-Net: Learning Elastic Properties of Deformable Objects from Graph Dynamicscited by 10
  3. 2025InstructionBench: An Instructional Video Understanding Benchmarkcited by 5
  4. 2021Comparing Reconstruction- and Contrastive-based Models for Visual Task Planningcited by 1