# Planning Immediate Landmarks of Targets for Model-Free Skill Transfer across Agents

arXiv: https://arxiv.org/abs/2212.09033 · HTML: https://mlsanity.com/paper/2212.09033/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 39 references that are in the map.
- First posted on arXiv 18 Dec 2022.

## Builds on (most cited first, 30 of 33)

- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [Data-Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1805.08296/) (2018)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [Learning to reinforcement learn](https://mlsanity.com/paper/1611.05763/) (2016)
- [Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates](https://mlsanity.com/paper/1610.00633/) (2016)
- [Successor Features for Transfer in Reinforcement Learning](https://mlsanity.com/paper/1606.05312/) (2016)
- [FeUdal Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1703.01161/) (2017)
- [Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation](https://mlsanity.com/paper/1604.06057/) (2016)
- [Transfer Learning in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2009.07888/) (2020)
- [Automatic Goal Generation for Reinforcement Learning Agents](https://mlsanity.com/paper/1705.06366/) (2017)
- [Contrastive Learning as Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2206.07568/) (2022)
- [Unity: A General Platform for Intelligent Agents](https://mlsanity.com/paper/1809.02627/) (2018)
- [DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills](https://mlsanity.com/paper/1804.02717/) (2018)
- [Goal-Conditioned Reinforcement Learning: Problems and Solutions](https://mlsanity.com/paper/2201.08299/) (2022)
- [One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control](https://mlsanity.com/paper/2007.04976/) (2020)
- [SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Driving](https://mlsanity.com/paper/2010.09776/) (2020)
- [Universal Successor Features Approximators](https://mlsanity.com/paper/1812.07626/) (2018)
- [Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learning](https://mlsanity.com/paper/2007.02832/) (2020)
- [MetaMorph: Learning Universal Controllers with Transformers](https://mlsanity.com/paper/2203.11931/) (2022)
- [Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learning](https://mlsanity.com/paper/1703.02949/) (2017)
- [Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewards](https://mlsanity.com/paper/1911.01417/) (2019)
- [My Body is a Cage: the Role of Morphology in Graph-Based Incompatible Control](https://mlsanity.com/paper/2010.01856/) (2020)
- [Learning and Transfer of Modulated Locomotor Controllers](https://mlsanity.com/paper/1610.05182/) (2016)
- [Hierarchical Reinforcement Learning By Discovering Intrinsic Options](https://mlsanity.com/paper/2101.06521/) (2021)
- [Sub-policy Adaptation for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1906.05862/) (2019)
- [Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2110.13625/) (2021)
- [DAC: The Double Actor-Critic Architecture for Learning Options](https://mlsanity.com/paper/1904.12691/) (2019)
- [Transfer RL across Observation Feature Spaces via Model-Based Regularization](https://mlsanity.com/paper/2201.00248/) (2022)
- [PerfectDou: Dominating DouDizhu with Perfect Information Distillation](https://mlsanity.com/paper/2203.16406/) (2022)

## Abstract (arXiv)

In reinforcement learning applications like robotics, agents usually need to deal with various input/output features when specified with different state/action spaces by their developers or physical restrictions. This indicates unnecessary re-training from scratch and considerable sample inefficiency, especially when agents follow similar solution steps to achieve tasks. In this paper, we aim to transfer similar high-level goal-transition knowledge to alleviate the challenge. Specifically, we propose PILoT, i.e., Planning Immediate Landmarks of Targets. PILoT utilizes the universal decoupled policy optimization to learn a goal-conditioned state planner; then, distills a goal-planner to plan immediate landmarks in a model-free style that can be shared among different agents. In our experiments, we show the power of PILoT on various transferring challenges, including few-shot transferring across action spaces and dynamics, from low-dimensional vector states to image inputs, from simple robot to complicated morphology; and we also illustrate a zero-shot transfer solution from a simple 2D navigation task to the harder Ant-Maze task.