# VPE: Variational Policy Embedding for Transfer Reinforcement Learning

arXiv: https://arxiv.org/abs/1809.03548 · HTML: https://mlsanity.com/paper/1809.03548/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Sep 2018.

## Built on by (most cited first, 8 of 8)

- [Multi-Agent Policy Transfer via Task Relationship Modeling](https://mlsanity.com/paper/2203.04482/) (2022)
- [Procedural Content Generation: Better Benchmarks for Transfer Reinforcement Learning](https://mlsanity.com/paper/2105.14780/) (2021)
- [Neural Stochastic Dual Dynamic Programming](https://mlsanity.com/paper/2112.00874/) (2021)
- [Prioritized Soft Q-Decomposition for Lexicographic Reinforcement Learning](https://mlsanity.com/paper/2310.02360/) (2023)
- [Learning Evolving Latent Strategies for Multi-Agent Language Systems without Model Fine-Tuning](https://mlsanity.com/paper/2512.20629/) (2025)
- [Reinforcement Learning in the Wild with Maximum Likelihood-based Model Transfer](https://mlsanity.com/paper/2302.09273/) (2023)
- [Learning Embeddings for Sequential Tasks Using Population of Agents](https://mlsanity.com/paper/2306.03311/) (2023)
- [Learning in Low-Dimensional Subspaces: Orthogonal Bottlenecks for Reinforcement Learning](https://mlsanity.com/paper/2605.26012/) (2026)

## Abstract (arXiv)

Reinforcement Learning methods are capable of solving complex problems, but resulting policies might perform poorly in environments that are even slightly different. In robotics especially, training and deployment conditions often vary and data collection is expensive, making retraining undesirable. Simulation training allows for feasible training times, but on the other hand suffers from a reality-gap when applied in real-world settings. This raises the need of efficient adaptation of policies acting in new environments. We consider this as a problem of transferring knowledge within a family of similar Markov decision processes. For this purpose we assume that Q-functions are generated by some low-dimensional latent variable. Given such a Q-function, we can find a master policy that can adapt given different values of this latent variable. Our method learns both the generative mapping and an approximate posterior of the latent variables, enabling identification of policies for new tasks by searching only in the latent space, rather than the space of all policies. The low-dimensional space, and master policy found by our method enables policies to quickly adapt to new environments. We demonstrate the method on both a pendulum swing-up task in simulation, and for simulation-to-real transfer on a pushing task.