Data to 14 Oct 2026

Self-Supervised Exploration via Disagreement

arXiv 1906.04161 ↗First posted 10 Jun 2019cs.LG cs.AI cs.CVcited by 265

Not on the map. Why?

Cited by 69 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Efficient exploration is a long-standing problem in sensorimotor learning. Major advances have been demonstrated in noise-free, non-stochastic domains such as video games and simulation. However, most of these formulations either get stuck in environments with stochastic dynamics or are too inefficient to be scalable to real robotics setups. In this paper, we propose a formulation for exploration inspired by the work in active learning literature. Specifically, we train an ensemble of dynamics models and incentivize the agent to explore such that the disagreement of those ensembles is maximized. This allows the agent to learn skills by exploring in a self-supervised manner without any external reward. Notably, we further leverage the disagreement objective to optimize the agent's policy in a differentiable manner, without using reinforcement learning, which results in a sample-efficient exploration. We demonstrate the efficacy of this formulation across a variety of benchmark environments including stochastic-Atari, Mujoco and Unity. Finally, we implement our differentiable exploration on a real robot which learns to interact with objects completely from scratch. Project videos and code are at https://pathak22.github.io/exploration-by-disagreement/

Key facts. Cited by 265 papers in the map (data through 14 Oct 2026). First posted on arXiv 10 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 50; 2022: 60; 2023: 63; 2024: 28; 2025: 33; 2026: 31.

265 papers.

Sort:

By area:

2021A Survey of Embodied AI: From Simulators to Research Taskscited by 261
2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 231
2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 174
2022Exploration in Deep Reinforcement Learning: A Surveycited by 166
2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 139
2021Discovering and Achieving Goals via World Modelscited by 131
2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 108
2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 108
2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 105
2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 96
2024Foundation Policies with Hilbert Representationscited by 94
2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 86

38 more

2021Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Taskscited by 78
2022BYOL-Explore: Exploration by Bootstrapped Predictioncited by 73
2021Causal Influence Detection for Improving Efficiency in Reinforcement Learningcited by 71
2022CIC: Contrastive Intrinsic Control for Unsupervised Skill Discoverycited by 68
2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learningcited by 61
2022Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learningcited by 60
2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 58
2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensemblescited by 56
2022Semantic Exploration from Language Abstractions and Pretrained Representationscited by 51
2021Learning to Map for Active Semantic Goal Navigationcited by 50
2023Controllability-Aware Unsupervised Skill Discoverycited by 49
2022Learning Agile Skills via Adversarial Imitation of Rough Partial Demonstrationscited by 45
2021MobILE: Model-Based Imitation Learning From Observation Alonecited by 44
2021Reducing Label Effort: Self-Supervised meets Active Learningcited by 42
2021MADE: Exploration via Maximizing Deviation from Explored Regionscited by 40
2021Learning more skills through optimistic explorationcited by 38
2023What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?cited by 36
2026Maximum Likelihood Reinforcement Learningcited by 36
2022Uncertainty-driven Planner for Exploration and Navigationcited by 33
2023Emergent Agentic Transformer from Chain of Hindsight Experiencecited by 33
2023Optimistic Active Exploration of Dynamical Systemscited by 32
2022Curious Exploration via Structured World Models Yields Zero-Shot Object Manipulationcited by 31
2022Choreographer: Learning and Adapting Skills in Imaginationcited by 31
2025Training a Generally Curious Agentcited by 31
2024MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximizationcited by 30
2021Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Surveycited by 28
2022An information-theoretic perspective on intrinsic motivation in reinforcement learning: a surveycited by 28
2024ASID: Active Exploration for System Identification in Robotic Manipulationcited by 28
2021How to Stay Curious while Avoiding Noisy TVs using Aleatoric Uncertainty Estimationcited by 27
2021Principled Exploration via Optimistic Bootstrapping and Backward Inductioncited by 26
2021Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coveragecited by 26
2022The Challenges of Exploration for Offline Reinforcement Learningcited by 26
2023Behavior Contrastive Learning for Unsupervised Skill Discoverycited by 26
2021Flexible Option Learningcited by 25
2022Minimax Optimal Online Imitation Learning via Replay Estimationcited by 25
2023Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shapingcited by 25
2025Reward Models in Deep Reinforcement Learning: A Surveycited by 25
2022The Free Energy Principle for Perception and Action: A Deep Learning Perspectivecited by 24

50 most cited of 265 All 265 (JSON)