# Fast Adaptation via Policy-Dynamics Value Functions

arXiv: https://arxiv.org/abs/2007.02879 · HTML: https://mlsanity.com/paper/2007.02879/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jul 2020.

## Built on by (most cited first, 13 of 13)

- [Decoupling Value and Policy for Generalization in Reinforcement Learning](https://mlsanity.com/paper/2102.10330/) (2021)
- [Learning Useful Representations of Recurrent Neural Network Weight Matrices](https://mlsanity.com/paper/2403.11998/) (2024)
- [One After Another: Learning Incremental Skills for a Changing World](https://mlsanity.com/paper/2203.11176/) (2022)
- [Composite Flow Matching for Reinforcement Learning with Shifted-Dynamics Data](https://mlsanity.com/paper/2505.23062/) (2025)
- [Fast Peer Adaptation with Context-aware Exploration](https://mlsanity.com/paper/2402.02468/) (2024)
- [OMPO: A Unified Framework for RL under Policy and Dynamics Shifts](https://mlsanity.com/paper/2405.19080/) (2024)
- [Towards A Unified Policy Abstraction Theory and Representation Learning Approach in Markov Decision Processes](https://mlsanity.com/paper/2209.07696/) (2022)
- [Provably Improved Context-Based Offline Meta-RL with Attention and Contrastive Learning](https://mlsanity.com/paper/2102.10774/) (2021)
- [SplAgger: Split Aggregation for Meta-Reinforcement Learning](https://mlsanity.com/paper/2403.03020/) (2024)
- [Learning Policy Representations for Steerable Behavior Synthesis](https://mlsanity.com/paper/2601.22350/) (2026)
- [Enhancing Hardware Fault Tolerance in Machines with Reinforcement Learning Policy Gradient Algorithms](https://mlsanity.com/paper/2407.15283/) (2024)
- [Curriculum reinforcement learning with measurable task representation learning](https://mlsanity.com/paper/2605.23372/) (2026)
- [When Does Test-Time Physical Diagnosis Pay? A Frozen Policy Buys Evidence It Never Reads](https://mlsanity.com/paper/2609.22299/) (2026)

## Abstract (arXiv)

Standard RL algorithms assume fixed environment dynamics and require a significant amount of interaction to adapt to new environments. We introduce Policy-Dynamics Value Functions (PD-VF), a novel approach for rapidly adapting to dynamics different from those previously seen in training. PD-VF explicitly estimates the cumulative reward in a space of policies and environments. An ensemble of conventional RL policies is used to gather experience on training environments, from which embeddings of both policies and environments can be learned. Then, a value function conditioned on both embeddings is trained. At test time, a few actions are sufficient to infer the environment embedding, enabling a policy to be selected by maximizing the learned value function (which requires no additional environment interaction). We show that our method can rapidly adapt to new dynamics on a set of MuJoCo domains. Code available at https://github.com/rraileanu/policy-dynamics-value-functions.