# Hypermodels for Exploration

arXiv: https://arxiv.org/abs/2006.07464 · HTML: https://mlsanity.com/paper/2006.07464/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jun 2020.

## Built on by (most cited first, 30 of 37)

- [Epistemic Neural Networks](https://mlsanity.com/paper/2107.08924/) (2021)
- [Reinforcement Learning, Bit by Bit](https://mlsanity.com/paper/2103.04047/) (2021)
- [Posterior Meta-Replay for Continual Learning](https://mlsanity.com/paper/2103.01133/) (2021)
- [Efficient Exploration for LLMs](https://mlsanity.com/paper/2402.00396/) (2024)
- [Uncertainty Estimation for Language Reward Models](https://mlsanity.com/paper/2203.07472/) (2022)
- [Approximate Thompson Sampling via Epistemic Neural Networks](https://mlsanity.com/paper/2302.09205/) (2023)
- [An Analysis of Ensemble Sampling](https://mlsanity.com/paper/2203.01303/) (2022)
- [Deciding What to Learn: A Rate-Distortion Approach](https://mlsanity.com/paper/2101.06197/) (2021)
- [The Neural Testbed: Evaluating Joint Predictions](https://mlsanity.com/paper/2110.04629/) (2021)
- [Pareto Set Learning for Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2501.06773/) (2025)
- [Deciding What to Model: Value-Equivalent Sampling for Reinforcement Learning](https://mlsanity.com/paper/2206.02072/) (2022)
- [Ensembles for Uncertainty Estimation: Benefits of Prior Functions and Bootstrapping](https://mlsanity.com/paper/2206.03633/) (2022)
- [Toward Efficient Exploration by Large Language Model Agents](https://mlsanity.com/paper/2504.20997/) (2025)
- [Sample-Efficient Alignment for LLMs](https://mlsanity.com/paper/2411.01493/) (2024)
- [The Value of Information When Deciding What to Learn](https://mlsanity.com/paper/2110.13973/) (2021)
- [Visual Affordance Prediction for Guiding Robot Exploration](https://mlsanity.com/paper/2305.17783/) (2023)
- [Scalable Neural Contextual Bandit for Recommender Systems](https://mlsanity.com/paper/2306.14834/) (2023)
- [Evaluating High-Order Predictive Distributions in Deep Learning](https://mlsanity.com/paper/2202.13509/) (2022)
- [Shattering the Agent-Environment Interface for Fine-Tuning Inclusive Language Models](https://mlsanity.com/paper/2305.11455/) (2023)
- [Variational Neural Networks](https://mlsanity.com/paper/2207.01524/) (2022)
- [Ensemble sampling for linear bandits: small ensembles suffice](https://mlsanity.com/paper/2311.08376/) (2023)
- [On Rate-Distortion Theory in Capacity-Limited Cognition & Reinforcement Learning](https://mlsanity.com/paper/2210.16877/) (2022)
- [Did we personalize? Assessing personalization by an online reinforcement learning algorithm using resampling](https://mlsanity.com/paper/2304.05365/) (2023)
- [Q-Star Meets Scalable Posterior Sampling: Bridging Theory and Practice via HyperAgent](https://mlsanity.com/paper/2402.10228/) (2024)
- [Random Latent Exploration for Deep Reinforcement Learning](https://mlsanity.com/paper/2407.13755/) (2024)
- [Contextual Thompson Sampling via Generation of Missing Data](https://mlsanity.com/paper/2502.07064/) (2025)
- [Ensemble Value Functions for Efficient Exploration in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2302.03439/) (2023)
- [Posterior Sampling for Continuing Environments](https://mlsanity.com/paper/2211.15931/) (2022)
- [Bayesian Reinforcement Learning with Limited Cognitive Load](https://mlsanity.com/paper/2305.03263/) (2023)
- [Satisficing Exploration for Deep Reinforcement Learning](https://mlsanity.com/paper/2407.12185/) (2024)

## Abstract (arXiv)

We study the use of hypermodels to represent epistemic uncertainty and guide exploration. This generalizes and extends the use of ensembles to approximate Thompson sampling. The computational cost of training an ensemble grows with its size, and as such, prior work has typically been limited to ensembles with tens of elements. We show that alternative hypermodels can enjoy dramatic efficiency gains, enabling behavior that would otherwise require hundreds or thousands of elements, and even succeed in situations where ensemble methods fail to learn regardless of size. This allows more accurate approximation of Thompson sampling as well as use of more sophisticated exploration schemes. In particular, we consider an approximate form of information-directed sampling and demonstrate performance gains relative to Thompson sampling. As alternatives to ensembles, we consider linear and neural network hypermodels, also known as hypernetworks. We prove that, with neural network base models, a linear hypermodel can represent essentially any distribution over functions, and as such, hypernetworks are no more expressive.