# Learning Robotic Manipulation through Visual Planning and Acting

arXiv: https://arxiv.org/abs/1905.04411 · HTML: https://mlsanity.com/paper/1905.04411/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 May 2019.

## Built on by (most cited first, 30 of 41)

- [VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objects](https://mlsanity.com/paper/2106.14440/) (2021)
- [RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks](https://mlsanity.com/paper/2205.02909/) (2022)
- [Unified Diffusion VLA: Vision-Language-Action Model via Joint Discrete Denoising Diffusion Process](https://mlsanity.com/paper/2511.01718/) (2025)
- [Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2204.13060/) (2022)
- [Temporal Straightening for Latent Planning](https://mlsanity.com/paper/2603.12231/) (2026)
- [Online Shielding for Reinforcement Learning](https://mlsanity.com/paper/2212.01861/) (2022)
- [Transfer Learning in Robotics: An Upcoming Breakthrough? A Review of Promises and Challenges](https://mlsanity.com/paper/2311.18044/) (2023)
- [Cross-Embodiment Robot Manipulation Skill Transfer using Latent Space Alignment](https://mlsanity.com/paper/2406.01968/) (2024)
- [Contact Points Discovery for Soft-Body Manipulations with Differentiable Physics](https://mlsanity.com/paper/2205.02835/) (2022)
- [Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks](https://mlsanity.com/paper/2109.10312/) (2021)
- [Learning Visually Guided Latent Actions for Assistive Teleoperation](https://mlsanity.com/paper/2105.00580/) (2021)
- [KEMP: Keyframe-Based Hierarchical End-to-End Deep Model for Long-Term Trajectory Prediction](https://mlsanity.com/paper/2205.04624/) (2022)
- [Autonomously Untangling Long Cables](https://mlsanity.com/paper/2207.07813/) (2022)
- [Efficient Self-Supervised Data Collection for Offline Robot Learning](https://mlsanity.com/paper/2105.04607/) (2021)
- [Deep Reinforcement Learning Based on Local GNN for Goal-conditioned Deformable Object Rearranging](https://mlsanity.com/paper/2302.10446/) (2023)
- [TC-IDM: Grounding Video Generation for Executable Zero-shot Robot Motion](https://mlsanity.com/paper/2601.18323/) (2026)
- [Domain-Adversarial and Conditional State Space Model for Imitation Learning](https://mlsanity.com/paper/2001.11628/) (2020)
- [In-air Knotting of Rope using Dual-Arm Robot based on Deep Learning](https://mlsanity.com/paper/2103.09402/) (2021)
- [Deformable Linear Object Prediction Using Locally Linear Latent Dynamics](https://mlsanity.com/paper/2103.14184/) (2021)
- [OSCAR: Data-Driven Operational Space Control for Adaptive and Robust Robot Manipulation](https://mlsanity.com/paper/2110.00704/) (2021)
- [Giving Robots a Hand: Learning Generalizable Manipulation with Eye-in-Hand Human Video Demonstrations](https://mlsanity.com/paper/2307.05959/) (2023)
- [SGTM 2.0: Autonomously Untangling Long Cables using Interactive Perception](https://mlsanity.com/paper/2209.13706/) (2022)
- [Discovering Closed-Loop Failures of Vision-Based Controllers via Reachability Analysis](https://mlsanity.com/paper/2211.02736/) (2022)
- [GOATS: Goal Sampling Adaptation for Scooping with Curriculum Reinforcement Learning](https://mlsanity.com/paper/2303.05193/) (2023)
- [Enabling Visual Action Planning for Object Manipulation through Latent Space Roadmap](https://mlsanity.com/paper/2103.02554/) (2021)
- [Predictive World Models from Real-World Partial Observations](https://mlsanity.com/paper/2301.04783/) (2023)
- [Unsupervised Feature Learning for Manipulation with Contrastive Domain Randomization](https://mlsanity.com/paper/2103.11144/) (2021)
- [V-MAO: Generative Modeling for Multi-Arm Manipulation of Articulated Objects](https://mlsanity.com/paper/2111.03987/) (2021)
- [Efficient Spatial Representation and Routing of Deformable One-Dimensional Objects for Manipulation](https://mlsanity.com/paper/2202.06172/) (2022)
- [Tactile-Sensitive NewtonianVAE for High-Accuracy Industrial Connector Insertion](https://mlsanity.com/paper/2203.05955/) (2022)

## Abstract (arXiv)

Planning for robotic manipulation requires reasoning about the changes a robot can affect on objects. When such interactions can be modelled analytically, as in domains with rigid objects, efficient planning algorithms exist. However, in both domestic and industrial domains, the objects of interest can be soft, or deformable, and hard to model analytically. For such cases, we posit that a data-driven modelling approach is more suitable. In recent years, progress in deep generative models has produced methods that learn to `imagine' plausible images from data. Building on the recent Causal InfoGAN generative model, in this work we learn to imagine goal-directed object manipulation directly from raw image data of self-supervised interaction of the robot with the object. After learning, given a goal observation of the system, our model can generate an imagined plan -- a sequence of images that transition the object into the desired goal. To execute the plan, we use it as a reference trajectory to track with a visual servoing controller, which we also learn from the data as an inverse dynamics model. In a simulated manipulation task, we show that separating the problem into visual planning and visual tracking control is more sample efficient and more interpretable than alternative data-driven approaches. We further demonstrate our approach on learning to imagine and execute in 3 environments, the final of which is deformable rope manipulation on a PR2 robot.