# Learn Dynamic-Aware State Embedding for Transfer Learning

arXiv: https://arxiv.org/abs/2101.02230 · HTML: https://mlsanity.com/paper/2101.02230/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 23 references that are in the map.
- First posted on arXiv 6 Jan 2021.

## Builds on (most cited first, 13 of 13)

- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [node2vec: Scalable Feature Learning for Networks](https://mlsanity.com/paper/1607.00653/) (2016)
- [DeepWalk: Online Learning of Social Representations](https://mlsanity.com/paper/1403.6652/) (2014)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [Successor Features for Transfer in Reinforcement Learning](https://mlsanity.com/paper/1606.05312/) (2016)
- [Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection](https://mlsanity.com/paper/1603.02199/) (2016)
- [Never Give Up: Learning Directed Exploration Strategies](https://mlsanity.com/paper/2002.06038/) (2020)
- [Novelty Search in Representational Space for Sample Efficient Exploration](https://mlsanity.com/paper/2009.13579/) (2020)
- [Universal Successor Features for Transfer Reinforcement Learning](https://mlsanity.com/paper/2001.04025/) (2020)
- [Advantages and Limitations of using Successor Features for Transfer in Reinforcement Learning](https://mlsanity.com/paper/1708.00102/) (2017)
- [Representation Learning on Graphs: A Reinforcement Learning Application](https://mlsanity.com/paper/1901.05351/) (2019)
- [Better transfer learning with inferred successor maps](https://mlsanity.com/paper/1906.07663/) (2019)

## Built on by (most cited first, 1 of 1)

- [A Research Agenda for Usability and Generalisation in Reinforcement Learning](https://mlsanity.com/paper/2412.16970/) (2024)

## Abstract (arXiv)

Transfer reinforcement learning aims to improve the sample efficiency of solving unseen new tasks by leveraging experiences obtained from previous tasks. We consider the setting where all tasks (MDPs) share the same environment dynamic except reward function. In this setting, the MDP dynamic is a good knowledge to transfer, which can be inferred by uniformly random policy. However, trajectories generated by uniform random policy are not useful for policy improvement, which impairs the sample efficiency severely. Instead, we observe that the binary MDP dynamic can be inferred from trajectories of any policy which avoids the need of uniform random policy. As the binary MDP dynamic contains the state structure shared over all tasks we believe it is suitable to transfer. Built on this observation, we introduce a method to infer the binary MDP dynamic on-line and at the same time utilize it to guide state embedding learning, which is then transferred to new tasks. We keep state embedding learning and policy learning separately. As a result, the learned state embedding is task and policy agnostic which makes it ideal for transfer learning. In addition, to facilitate the exploration over the state space, we propose a novel intrinsic reward based on the inferred binary MDP dynamic. Our method can be used out-of-box in combination with model-free RL algorithms. We show two instances on the basis of algoDQN and algoA2C. Empirical results of intensive experiments show the advantage of our proposed method in various transfer learning tasks.