# Dynamics Learning with Cascaded Variational Inference for Multi-Step Manipulation

arXiv: https://arxiv.org/abs/1910.13395 · HTML: https://mlsanity.com/paper/1910.13395/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Oct 2019.

## Built on by (most cited first, 24 of 24)

- [ManipulaTHOR: A Framework for Visual Object Manipulation](https://mlsanity.com/paper/2104.11213/) (2021)
- [Latent Plans for Task-Agnostic Offline Reinforcement Learning](https://mlsanity.com/paper/2209.08959/) (2022)
- [Deep Imitation Learning for Bimanual Robotic Manipulation](https://mlsanity.com/paper/2010.05134/) (2020)
- [Hierarchical Planning with Latent World Models](https://mlsanity.com/paper/2604.03208/) (2026)
- [LASER: Learning a Latent Action Space for Efficient Reinforcement Learning](https://mlsanity.com/paper/2103.15793/) (2021)
- [Planning to Practice: Efficient Online Fine-Tuning by Composing Goals in Latent Space](https://mlsanity.com/paper/2205.08129/) (2022)
- [Generalization with Lossy Affordances: Leveraging Broad Offline Data for Learning Visuomotor Tasks](https://mlsanity.com/paper/2210.06601/) (2022)
- [DexDeform: Dexterous Deformable Object Manipulation with Human Demonstrations and Differentiable Physics](https://mlsanity.com/paper/2304.03223/) (2023)
- [Subgoal Search For Complex Reasoning Tasks](https://mlsanity.com/paper/2108.11204/) (2021)
- [Reparameterized Policy Learning for Multimodal Trajectory Optimization](https://mlsanity.com/paper/2307.10710/) (2023)
- [Temporal Representation Alignment: Successor Features Enable Emergent Compositionality in Robot Instruction Following](https://mlsanity.com/paper/2502.05454/) (2025)
- [Closing the Planning-Learning Loop with Application to Autonomous Driving](https://mlsanity.com/paper/2101.03834/) (2021)
- [Fast and Precise: Adjusting Planning Horizon with Adaptive Subgoal Search](https://mlsanity.com/paper/2206.00702/) (2022)
- [Action Dynamics Task Graphs for Learning Plannable Representations of Procedural Tasks](https://mlsanity.com/paper/2302.05330/) (2023)
- [ActionDiffusion: An Action-aware Diffusion Model for Procedure Planning in Instructional Videos](https://mlsanity.com/paper/2403.08591/) (2024)
- [Hybrid Search for Efficient Planning with Completeness Guarantees](https://mlsanity.com/paper/2310.12819/) (2023)
- [DiffVL: Scaling Up Soft Body Manipulation using Vision-Language Driven Differentiable Physics](https://mlsanity.com/paper/2312.06408/) (2023)
- [Discovering Generalizable Skills via Automated Generation of Diverse Tasks](https://mlsanity.com/paper/2106.13935/) (2021)
- [Active Task Randomization: Learning Robust Skills via Unsupervised Generation of Diverse and Feasible Tasks](https://mlsanity.com/paper/2211.06134/) (2022)
- [STEADY: Simultaneous State Estimation and Dynamics Learning from Indirect Observations](https://mlsanity.com/paper/2203.01299/) (2022)
- [Unifying Object-Centric World Models and Diffusion Policy: A Hierarchical Framework for Multi-Stage Robotic Tasks](https://mlsanity.com/paper/2606.08775/) (2026)
- [Correspondence-Oriented Imitation Learning: Flexible Visuomotor Control with 3D Conditioning](https://mlsanity.com/paper/2512.05953/) (2025)
- [Flow Policies as Actions of Skill-Level World Models: Learned and Symbolic Abstractions for Long-Horizon Planning](https://mlsanity.com/paper/2610.04767/) (2026)
- [H-JEPA: End-to-End Learning of Hierarchical World Models for Visual Planning](https://mlsanity.com/paper/2610.06805/) (2026)

## Abstract (arXiv)

The fundamental challenge of planning for multi-step manipulation is to find effective and plausible action sequences that lead to the task goal. We present Cascaded Variational Inference (CAVIN) Planner, a model-based method that hierarchically generates plans by sampling from latent spaces. To facilitate planning over long time horizons, our method learns latent representations that decouple the prediction of high-level effects from the generation of low-level motions through cascaded variational inference. This enables us to model dynamics at two different levels of temporal resolutions for hierarchical planning. We evaluate our approach in three multi-step robotic manipulation tasks in cluttered tabletop environments given high-dimensional observations. Empirical results demonstrate that the proposed method outperforms state-of-the-art model-based methods by strategically interacting with multiple objects.