# Universal Planning Networks

arXiv: https://arxiv.org/abs/1804.00645 · HTML: https://mlsanity.com/paper/1804.00645/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Apr 2018.

## Built on by (most cited first, 30 of 30)

- [Contrastive Learning as Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2206.07568/) (2022)
- [Procedure Planning in Instructional Videos via Contextual Modeling and Model-based Policy Learning](https://mlsanity.com/paper/2110.01770/) (2021)
- [Differentiable Spatial Planning using Transformers](https://mlsanity.com/paper/2112.01010/) (2021)
- [Bridging Imagination and Reality for Model-Based Deep Reinforcement Learning](https://mlsanity.com/paper/2010.12142/) (2020)
- [Zero-shot Imitation Learning from Demonstrations for Legged Robot Visual Navigation](https://mlsanity.com/paper/1909.12971/) (2019)
- [Motion Planning by Learning the Solution Manifold in Trajectory Optimization](https://mlsanity.com/paper/2107.05842/) (2021)
- [High-Accuracy Model-Based Reinforcement Learning, a Survey](https://mlsanity.com/paper/2107.08241/) (2021)
- [Diffused Task-Agnostic Milestone Planner](https://mlsanity.com/paper/2312.03395/) (2023)
- [Hierarchies of Planning and Reinforcement Learning for Robot Navigation](https://mlsanity.com/paper/2109.11178/) (2021)
- [To Follow or not to Follow: Selective Imitation Learning from Observations](https://mlsanity.com/paper/1912.07670/) (2019)
- [LS3: Latent Space Safe Sets for Long-Horizon Visuomotor Control of Sparse Reward Iterative Tasks](https://mlsanity.com/paper/2107.04775/) (2021)
- [PathBench: A Benchmarking Platform for Classical and Learned Path Planning Algorithms](https://mlsanity.com/paper/2105.01777/) (2021)
- [Reinforcement Learning in Practice: Opportunities and Challenges](https://mlsanity.com/paper/2202.11296/) (2022)
- [Integrating Symmetry into Differentiable Planning with Steerable Convolutions](https://mlsanity.com/paper/2206.03674/) (2022)
- [Scaling up and Stabilizing Differentiable Planning with Implicit Differentiation](https://mlsanity.com/paper/2210.13542/) (2022)
- [Enabling Visual Action Planning for Object Manipulation through Latent Space Roadmap](https://mlsanity.com/paper/2103.02554/) (2021)
- [Waypoint Planning Networks](https://mlsanity.com/paper/2105.00312/) (2021)
- [Online Control-Informed Learning](https://mlsanity.com/paper/2410.03924/) (2024)
- [Improving Model-Based Reinforcement Learning with Internal State Representations through Self-Supervision](https://mlsanity.com/paper/2102.05599/) (2021)
- [Next Steps: Learning a Disentangled Gait Representation for Versatile Quadruped Locomotion](https://mlsanity.com/paper/2112.04809/) (2021)
- [VAE-Loco: Versatile Quadruped Locomotion by Learning a Disentangled Gait Representation](https://mlsanity.com/paper/2205.01179/) (2022)
- [Introspective Visuomotor Control: Exploiting Uncertainty in Deep Visuomotor Control for Failure Recovery](https://mlsanity.com/paper/2103.11881/) (2021)
- [CLAMGen: Closed-Loop Arm Motion Generation via Multi-view Vision-Based RL](https://mlsanity.com/paper/2103.13267/) (2021)
- [Know Thyself: Transferable Visual Control Policies Through Robot-Awareness](https://mlsanity.com/paper/2107.09047/) (2021)
- [Safe Online Control-Informed Learning](https://mlsanity.com/paper/2512.13868/) (2025)
- [In-Context Learning for Robots: Methods and Applications](https://mlsanity.com/paper/2609.36012/) (2026)
- [Visual Goal-Directed Meta-Learning with Contextual Planning Networks](https://mlsanity.com/paper/2111.09908/) (2021)
- [Systematic Comparison of Path Planning Algorithms using PathBench](https://mlsanity.com/paper/2203.03092/) (2022)
- [Visual processing in context of reinforcement learning](https://mlsanity.com/paper/2208.12525/) (2022)
- [STRIPS-WM: Learning Grounded Propositional STRIPS-style World Models from Images](https://mlsanity.com/paper/2606.06832/) (2026)

## Abstract (arXiv)

A key challenge in complex visuomotor control is learning abstract representations that are effective for specifying goals, planning, and generalization. To this end, we introduce universal planning networks (UPN). UPNs embed differentiable planning within a goal-directed policy. This planning computation unrolls a forward model in a latent space and infers an optimal action plan through gradient descent trajectory optimization. The plan-by-gradient-descent process and its underlying representations are learned end-to-end to directly optimize a supervised imitation learning objective. We find that the representations learned are not only effective for goal-directed visual imitation via gradient-based trajectory optimization, but can also provide a metric for specifying goals using images. The learned representations can be leveraged to specify distance-based rewards to reach new target states for model-free reinforcement learning, resulting in substantially more effective learning when solving new tasks described via image-based goals. We were able to achieve successful transfer of visuomotor planning strategies across robots with significantly different morphologies and actuation capabilities.