# Interesting Object, Curious Agent: Learning Task-Agnostic Exploration

arXiv: https://arxiv.org/abs/2111.13119 · HTML: https://mlsanity.com/paper/2111.13119/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 77 references that are in the map.
- First posted on arXiv 25 Nov 2021.

## Builds on (most cited first, 30 of 32)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [Progress & Compress: A scalable framework for continual learning](https://mlsanity.com/paper/1805.06370/) (2018)
- [Is Q-learning Provably Efficient?](https://mlsanity.com/paper/1807.03765/) (2018)
- [Policy Distillation](https://mlsanity.com/paper/1511.06295/) (2015)
- [Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables](https://mlsanity.com/paper/1903.08254/) (2019)
- [Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey](https://mlsanity.com/paper/2003.04960/) (2020)
- [Successor Features for Transfer in Reinforcement Learning](https://mlsanity.com/paper/1606.05312/) (2016)
- [#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning](https://mlsanity.com/paper/1611.04717/) (2016)
- [Tackling Climate Change with Machine Learning](https://mlsanity.com/paper/1906.05433/) (2019)
- [Count-Based Exploration with Neural Density Models](https://mlsanity.com/paper/1703.01310/) (2017)
- [Distral: Robust Multitask Reinforcement Learning](https://mlsanity.com/paper/1707.04175/) (2017)
- [Neural Topological SLAM for Visual Navigation](https://mlsanity.com/paper/2005.12256/) (2020)
- [Reinforcement Learning with Prototypical Representations](https://mlsanity.com/paper/2102.11271/) (2021)
- [Deep Exploration via Randomized Value Functions](https://mlsanity.com/paper/1703.07608/) (2017)
- [Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models](https://mlsanity.com/paper/1507.00814/) (2015)
- [RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments](https://mlsanity.com/paper/2002.12292/) (2020)
- [Fast Task Inference with Variational Intrinsic Successor Features](https://mlsanity.com/paper/1906.05030/) (2019)
- [Q-learning with UCB Exploration is Sample Efficient for Infinite-Horizon MDP](https://mlsanity.com/paper/1901.09311/) (2019)
- [Assistive Gym: A Physics Simulation Framework for Assistive Robotics](https://mlsanity.com/paper/1910.04700/) (2019)
- [TorchBeast: A PyTorch Platform for Distributed RL](https://mlsanity.com/paper/1910.03552/) (2019)
- [Investigating Human Priors for Playing Video Games](https://mlsanity.com/paper/1802.10217/) (2018)

## Built on by (most cited first, 30 of 38)

- [Affordances from Human Videos as a Versatile Representation for Robotics](https://mlsanity.com/paper/2304.08488/) (2023)
- [CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation](https://mlsanity.com/paper/2203.10421/) (2022)
- [The Unsurprising Effectiveness of Pre-Trained Vision Models for Control](https://mlsanity.com/paper/2203.03580/) (2022)
- [RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulation](https://mlsanity.com/paper/2402.15487/) (2024)
- [Choreographer: Learning and Adapting Skills in Imagination](https://mlsanity.com/paper/2211.13350/) (2022)
- [The Challenges of Exploration for Offline Reinforcement Learning](https://mlsanity.com/paper/2201.11861/) (2022)
- [A Study of Global and Episodic Bonuses for Exploration in Contextual MDPs](https://mlsanity.com/paper/2306.03236/) (2023)
- [NAVIX: Scaling MiniGrid Environments with JAX](https://mlsanity.com/paper/2407.19396/) (2024)
- [Unsupervised Reinforcement Learning in Multiple Environments](https://mlsanity.com/paper/2112.08746/) (2021)
- [Curious Replay for Model-based Adaptation](https://mlsanity.com/paper/2306.15934/) (2023)
- [Episodic Novelty Through Temporal Distance](https://mlsanity.com/paper/2501.15418/) (2025)
- [Efficient RL via Disentangled Environment and Agent Representations](https://mlsanity.com/paper/2309.02435/) (2023)
- [Exploring through Random Curiosity with General Value Functions](https://mlsanity.com/paper/2211.10282/) (2022)
- [Monitored Markov Decision Processes](https://mlsanity.com/paper/2402.06819/) (2024)
- [Exploitation-Guided Exploration for Semantic Embodied Navigation](https://mlsanity.com/paper/2311.03357/) (2023)
- [From Text to Life: On the Reciprocal Relationship between Artificial Life and Large Language Models](https://mlsanity.com/paper/2407.09502/) (2024)
- [Self-Motivated Multi-Agent Exploration](https://mlsanity.com/paper/2301.02083/) (2023)
- [Beyond Optimism: Exploration With Partially Observable Rewards](https://mlsanity.com/paper/2406.13909/) (2024)
- [SOE: Sample-Efficient Robot Policy Self-Improvement via On-Manifold Exploration](https://mlsanity.com/paper/2509.19292/) (2025)
- [Approximate information state based convergence analysis of recurrent Q-learning](https://mlsanity.com/paper/2306.05991/) (2023)
- [CuriousBot: Interactive Mobile Exploration via Actionable 3D Relational Object Graph](https://mlsanity.com/paper/2501.13338/) (2025)
- [SFP: State-free Priors for Exploration in Off-Policy Reinforcement Learning](https://mlsanity.com/paper/2205.13528/) (2022)
- [Go Beyond Imagination: Maximizing Episodic Reachability with World Models](https://mlsanity.com/paper/2308.13661/) (2023)
- [Just Cluster It: An Approach for Exploration in High-Dimensions using Clustering and Pre-Trained Representations](https://mlsanity.com/paper/2402.03138/) (2024)
- [ExToken: Structured Exploration for Efficient Vision-Language-Action Reinforcement Fine-tuning](https://mlsanity.com/paper/2607.12931/) (2026)
- [Human-Inspired Framework to Accelerate Reinforcement Learning](https://mlsanity.com/paper/2303.08115/) (2023)
- [Cyclophobic Reinforcement Learning](https://mlsanity.com/paper/2308.15911/) (2023)
- [Training on Irrelevant States Implies Data Augmentation: Generalization in Contextual MDPs](https://mlsanity.com/paper/2410.03565/) (2024)
- [Embodied Crowd Counting](https://mlsanity.com/paper/2503.08367/) (2025)
- [World Model Agents with Change-Based Intrinsic Motivation](https://mlsanity.com/paper/2503.21047/) (2025)

## Abstract (arXiv)

Common approaches for task-agnostic exploration learn tabula-rasa --the agent assumes isolated environments and no prior knowledge or experience. However, in the real world, agents learn in many environments and always come with prior experiences as they explore new ones. Exploration is a lifelong process. In this paper, we propose a paradigm change in the formulation and evaluation of task-agnostic exploration. In this setup, the agent first learns to explore across many environments without any extrinsic goal in a task-agnostic manner. Later on, the agent effectively transfers the learned exploration policy to better explore new environments when solving tasks. In this context, we evaluate several baseline exploration strategies and present a simple yet effective approach to learning task-agnostic exploration policies. Our key idea is that there are two components of exploration: (1) an agent-centric component encouraging exploration of unseen parts of the environment based on an agent's belief; (2) an environment-centric component encouraging exploration of inherently interesting objects. We show that our formulation is effective and provides the most consistent exploration across several training-testing environment pairs. We also introduce benchmarks and metrics for evaluating task-agnostic exploration strategies. The source code is available at https://github.com/sparisi/cbet/.