# Visual Goal-Directed Meta-Learning with Contextual Planning Networks

arXiv: https://arxiv.org/abs/2111.09908 · HTML: https://mlsanity.com/paper/2111.09908/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 33 references that are in the map.
- First posted on arXiv 18 Nov 2021.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Generalizing from a Few Examples: A Survey on Few-Shot Learning](https://mlsanity.com/paper/1904.05046/) (2019)
- [Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables](https://mlsanity.com/paper/1903.08254/) (2019)
- [Benchmarking Deep Reinforcement Learning for Continuous Control](https://mlsanity.com/paper/1604.06778/) (2016)
- [Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images](https://mlsanity.com/paper/1506.07365/) (2015)
- [Deep Spatial Autoencoders for Visuomotor Learning](https://mlsanity.com/paper/1509.06113/) (2015)
- [Value Prediction Network](https://mlsanity.com/paper/1707.03497/) (2017)
- [Zero-Shot Visual Imitation](https://mlsanity.com/paper/1804.08606/) (2018)
- [The Predictron: End-To-End Learning and Planning](https://mlsanity.com/paper/1612.08810/) (2016)
- [Combining Self-Supervised Learning and Imitation for Vision-Based Rope Manipulation](https://mlsanity.com/paper/1703.02018/) (2017)
- [Learning to Continually Learn](https://mlsanity.com/paper/2002.09571/) (2020)
- [Neural Task Programming: Learning to Generalize Across Hierarchical Tasks](https://mlsanity.com/paper/1710.01813/) (2017)
- [Unsupervised Perceptual Rewards for Imitation Learning](https://mlsanity.com/paper/1612.06699/) (2016)
- [TACO: Learning Task Decomposition via Temporal Alignment for Control](https://mlsanity.com/paper/1803.01840/) (2018)
- [Universal Planning Networks](https://mlsanity.com/paper/1804.00645/) (2018)
- [Watch, Try, Learn: Meta-Learning from Demonstrations and Reward](https://mlsanity.com/paper/1906.03352/) (2019)
- [Hallucinative Topological Memory for Zero-Shot Visual Planning](https://mlsanity.com/paper/2002.12336/) (2020)
- [Model-Based Planning with Discrete and Continuous Actions](https://mlsanity.com/paper/1705.07177/) (2017)

## Abstract (arXiv)

The goal of meta-learning is to generalize to new tasks and goals as quickly as possible. Ideally, we would like approaches that generalize to new goals and tasks on the first attempt. Toward that end, we introduce contextual planning networks (CPN). Tasks are represented as goal images and used to condition the approach. We evaluate CPN along with several other approaches adapted for zero-shot goal-directed meta-learning. We evaluate these approaches across 24 distinct manipulation tasks using Metaworld benchmark tasks. We found that CPN outperformed several approaches and baselines on one task and was competitive with existing approaches on others. We demonstrate the approach on a physical platform on Jenga tasks using a Kinova Jaco robotic arm.