# EX2: Exploration with Exemplar Models for Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/1703.01260 · HTML: https://mlsanity.com/paper/1703.01260/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Mar 2017.

## Built on by (most cited first, 30 of 42)

- [How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned](https://mlsanity.com/paper/2102.02915/) (2021)
- [Exploration in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2205.00824/) (2022)
- [Human-to-Robot Imitation in the Wild](https://mlsanity.com/paper/2207.09450/) (2022)
- [METRA: Scalable Unsupervised RL with Metric-Aware Abstraction](https://mlsanity.com/paper/2310.08887/) (2023)
- [A Survey of Exploration Methods in Reinforcement Learning](https://mlsanity.com/paper/2109.00157/) (2021)
- [Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain](https://mlsanity.com/paper/2109.06668/) (2021)
- [Cooperative Exploration for Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/2107.11444/) (2021)
- [An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey](https://mlsanity.com/paper/2209.08890/) (2022)
- [MURAL: Meta-Learning Uncertainty-Aware Rewards for Outcome-Driven Reinforcement Learning](https://mlsanity.com/paper/2107.07184/) (2021)
- [Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learning](https://mlsanity.com/paper/2408.03029/) (2024)
- [Model-based Offline Reinforcement Learning with Count-based Conservatism](https://mlsanity.com/paper/2307.11352/) (2023)
- [Off-Policy RL Algorithms Can be Sample-Efficient for Continuous Control via Sample Multiple Reuse](https://mlsanity.com/paper/2305.18443/) (2023)
- [Information is Power: Intrinsic Control via Information Capture](https://mlsanity.com/paper/2112.03899/) (2021)
- [Exploration-Driven Generative Interactive Environments](https://mlsanity.com/paper/2504.02515/) (2025)
- [Novelty Detection in Reinforcement Learning with World Models](https://mlsanity.com/paper/2310.08731/) (2023)
- [Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Bias](https://mlsanity.com/paper/2310.08558/) (2023)
- [From Psychological Curiosity to Artificial Curiosity: Curiosity-Driven Learning in Artificial Intelligence Tasks](https://mlsanity.com/paper/2201.08300/) (2022)
- [Successor-Predecessor Intrinsic Exploration](https://mlsanity.com/paper/2305.15277/) (2023)
- [Constrained Intrinsic Motivation for Reinforcement Learning](https://mlsanity.com/paper/2407.09247/) (2024)
- [Random Latent Exploration for Deep Reinforcement Learning](https://mlsanity.com/paper/2407.13755/) (2024)
- [Safety through feedback in Constrained RL](https://mlsanity.com/paper/2406.19626/) (2024)
- [Emergence of Exploration in Policy Gradient Reinforcement Learning via Retrying](https://mlsanity.com/paper/2606.00151/) (2026)
- [Curiosity in Hindsight: Intrinsic Exploration in Stochastic Environments](https://mlsanity.com/paper/2211.10515/) (2022)
- [ALP: Action-Aware Embodied Learning for Perception](https://mlsanity.com/paper/2306.10190/) (2023)
- [Intrinsically motivated graph exploration using network theories of human curiosity](https://mlsanity.com/paper/2307.04962/) (2023)
- [Image-Based Deep Reinforcement Learning with Intrinsically Motivated Stimuli: On the Execution of Complex Robotic Tasks](https://mlsanity.com/paper/2407.21338/) (2024)
- [Playtesting: What is Beyond Personas](https://mlsanity.com/paper/2107.11965/) (2021)
- [Generative Actor Critic](https://mlsanity.com/paper/2512.21527/) (2025)
- [3SPO: State-Score-Supervised Policy Optimization for LLM Agents](https://mlsanity.com/paper/2606.09961/) (2026)
- [Intrinsically Motivated Self-supervised Learning in Reinforcement Learning](https://mlsanity.com/paper/2106.13970/) (2021)

## Abstract (arXiv)

Deep reinforcement learning algorithms have been shown to learn complex tasks using highly general policy classes. However, sparse reward problems remain a significant challenge. Exploration methods based on novelty detection have been particularly successful in such settings but typically require generative or predictive models of the observations, which can be difficult to train when the observations are very high-dimensional and complex, as in the case of raw images. We propose a novelty detection algorithm for exploration that is based entirely on discriminatively trained exemplar models, where classifiers are trained to discriminate each visited state against all others. Intuitively, novel states are easier to distinguish against other states seen during training. We show that this kind of discriminative modeling corresponds to implicit density estimation, and that it can be combined with count-based exploration to produce competitive results on a range of popular benchmark tasks, including state-of-the-art results on challenging egocentric observations in the vizDoom benchmark.