# Learning Adaptive Exploration Strategies in Dynamic Environments Through Informed Policy Regularization

arXiv: https://arxiv.org/abs/2005.02934 · HTML: https://mlsanity.com/paper/2005.02934/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 May 2020.

## Built on by (most cited first, 16 of 16)

- [Offline Meta-Reinforcement Learning with Online Self-Supervision](https://mlsanity.com/paper/2107.03974/) (2021)
- [AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agents](https://mlsanity.com/paper/2310.09971/) (2023)
- [Block Contextual MDPs for Continual Learning](https://mlsanity.com/paper/2110.06972/) (2021)
- [Recurrent Hypernetworks are Surprisingly Strong in Meta-RL](https://mlsanity.com/paper/2309.14970/) (2023)
- [An Adaptive Deep RL Method for Non-Stationary Environments with Piecewise Stable Context](https://mlsanity.com/paper/2212.12735/) (2022)
- [Learning a subspace of policies for online adaptation in Reinforcement Learning](https://mlsanity.com/paper/2110.05169/) (2021)
- [MetaVIM: Meta Variationally Intrinsic Motivated Reinforcement Learning for Decentralized Traffic Signal Control](https://mlsanity.com/paper/2101.00746/) (2021)
- [Meta-Reinforcement Learning by Tracking Task Non-stationarity](https://mlsanity.com/paper/2105.08834/) (2021)
- [Meta Policy Learning for Cold-Start Conversational Recommendation](https://mlsanity.com/paper/2205.11788/) (2022)
- [Giving Feedback on Interactive Student Programs with Meta-Exploration](https://mlsanity.com/paper/2211.08802/) (2022)
- [Behavioral Exploration: Learning to Explore via In-Context Adaptation](https://mlsanity.com/paper/2507.09041/) (2025)
- [Simple Embodied Language Learning as a Byproduct of Meta-Reinforcement Learning](https://mlsanity.com/paper/2306.08400/) (2023)
- [Lifelong Reinforcement Learning via Neuromodulation](https://mlsanity.com/paper/2408.08446/) (2024)
- [Online Modeling and Monitoring of Dependent Processes under Resource Constraints](https://mlsanity.com/paper/2307.14208/) (2023)
- [In-Context Reinforcement Learning under Non-Stationarity: A Survey](https://mlsanity.com/paper/2607.11906/) (2026)
- [Towards a Reward-Free Reinforcement Learning Framework for Vehicle Control](https://mlsanity.com/paper/2502.15262/) (2025)

## Abstract (arXiv)

We study the problem of learning exploration-exploitation strategies that effectively adapt to dynamic environments, where the task may change over time. While RNN-based policies could in principle represent such strategies, in practice their training time is prohibitive and the learning process often converges to poor solutions. In this paper, we consider the case where the agent has access to a description of the task (e.g., a task id or task parameters) at training time, but not at test time. We propose a novel algorithm that regularizes the training of an RNN-based policy using informed policies trained to maximize the reward in each task. This dramatically reduces the sample complexity of training RNN-based policies, without losing their representational power. As a result, our method learns exploration strategies that efficiently balance between gathering information about the unknown and changing task and maximizing the reward over time. We test the performance of our algorithm in a variety of environments where tasks may vary within each episode.