# Exploration in Approximate Hyper-State Space for Meta Reinforcement Learning

arXiv: https://arxiv.org/abs/2010.01062 · HTML: https://mlsanity.com/paper/2010.01062/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2020.

## Built on by (most cited first, 28 of 28)

- [Automated Design of Agentic Systems](https://mlsanity.com/paper/2408.08435/) (2024)
- [A Survey of Zero-shot Generalisation in Deep Reinforcement Learning](https://mlsanity.com/paper/2111.09794/) (2021)
- [Bridging State and History Representations: Understanding Self-Predictive RL](https://mlsanity.com/paper/2401.08898/) (2024)
- [Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environment](https://mlsanity.com/paper/2104.05632/) (2021)
- [AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agents](https://mlsanity.com/paper/2310.09971/) (2023)
- [Dynamics Generalisation in Reinforcement Learning via Adaptive Context-Aware Policies](https://mlsanity.com/paper/2310.16686/) (2023)
- [On the Effectiveness of Fine-tuning Versus Meta-reinforcement Learning](https://mlsanity.com/paper/2206.03271/) (2022)
- [MAMBA: an Effective World Model Approach for Meta-Reinforcement Learning](https://mlsanity.com/paper/2403.09859/) (2024)
- [Distilling Reinforcement Learning Algorithms for In-Context Model-Based Planning](https://mlsanity.com/paper/2502.19009/) (2025)
- [Active Inference as a Model of Agency](https://mlsanity.com/paper/2401.12917/) (2024)
- [A CMDP-within-online framework for Meta-Safe Reinforcement Learning](https://mlsanity.com/paper/2405.16601/) (2024)
- [Distributionally Adaptive Meta Reinforcement Learning](https://mlsanity.com/paper/2210.03104/) (2022)
- [Meta-Reinforcement Learning Based on Self-Supervised Task Representation Learning](https://mlsanity.com/paper/2305.00286/) (2023)
- [ContraBAR: Contrastive Bayes-Adaptive Deep RL](https://mlsanity.com/paper/2306.02418/) (2023)
- [MetaVIM: Meta Variationally Intrinsic Motivated Reinforcement Learning for Decentralized Traffic Signal Control](https://mlsanity.com/paper/2101.00746/) (2021)
- [Planning to the Information Horizon of BAMDPs via Epistemic State Abstraction](https://mlsanity.com/paper/2210.16872/) (2022)
- [First-Explore, then Exploit: Meta-Learning to Solve Hard Exploration-Exploitation Trade-Offs](https://mlsanity.com/paper/2307.02276/) (2023)
- [Bayesian Exploration Networks](https://mlsanity.com/paper/2308.13049/) (2023)
- [Enabling Adaptive Agent Training in Open-Ended Simulators by Targeting Diversity](https://mlsanity.com/paper/2411.04466/) (2024)
- [BIMRL: Brain Inspired Meta Reinforcement Learning](https://mlsanity.com/paper/2210.16530/) (2022)
- [A Bayesian Solution To The Imitation Gap](https://mlsanity.com/paper/2407.00495/) (2024)
- [Never Explore Repeatedly in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2308.09909/) (2023)
- [Combining Bayesian Inference and Reinforcement Learning for Agent Decision Making: A Review](https://mlsanity.com/paper/2505.07911/) (2025)
- [Knowledge is reward: Learning optimal exploration by predictive reward cashing](https://mlsanity.com/paper/2109.08518/) (2021)
- [Reinforcement Learning for Few-Shot Text Generation Adaptation](https://mlsanity.com/paper/2111.11030/) (2021)
- [RL3: Boosting Meta Reinforcement Learning via RL inside RL2](https://mlsanity.com/paper/2306.15909/) (2023)
- [Learning Task Belief Similarity with Latent Dynamics for Meta-Reinforcement Learning](https://mlsanity.com/paper/2506.19785/) (2025)
- [Meta-Learning and Meta-Reinforcement Learning -- Tracing the Path towards DeepMind's Adaptive Agent](https://mlsanity.com/paper/2602.19837/) (2026)

## Abstract (arXiv)

To rapidly learn a new task, it is often essential for agents to explore efficiently -- especially when performance matters from the first timestep. One way to learn such behaviour is via meta-learning. Many existing methods however rely on dense rewards for meta-training, and can fail catastrophically if the rewards are sparse. Without a suitable reward signal, the need for exploration during meta-training is exacerbated. To address this, we propose HyperX, which uses novel reward bonuses for meta-training to explore in approximate hyper-state space (where hyper-states represent the environment state and the agent's task belief). We show empirically that HyperX meta-learns better task-exploration and adapts more successfully to new tasks than existing methods.