mlsanityData to 14 Oct 2026

A Study of Global and Episodic Bonuses for Exploration in Contextual MDPs

arXiv 2306.03236 ↗First posted 5 Jun 2023cs.AIcited by 20

Not on the map. Why?

Abstract (arXiv)

Exploration in environments which differ across episodes has received increasing attention in recent years. Current methods use some combination of global novelty bonuses, computed using the agent's entire training experience, and textitepisodic novelty bonuses, computed using only experience from the current episode. However, the use of these two types of bonuses has been ad-hoc and poorly understood. In this work, we shed light on the behavior of these two types of bonuses through controlled experiments on easily interpretable tasks as well as challenging pixel-based settings. We find that the two types of bonuses succeed in different settings, with episodic bonuses being most effective when there is little shared structure across episodes and global bonuses being effective when more structure is shared. We develop a conceptual framework which makes this notion of shared structure precise by considering the variance of the value function across contexts, and which provides a unifying explanation of our empirical results. We furthermore find that combining the two bonuses can lead to more robust performance across different degrees of shared structure, and investigate different algorithmic choices for defining and combining global and episodic bonuses based on function approximation. This results in an algorithm which sets a new state of the art across 16 tasks from the MiniHack suite used in prior work, and also performs robustly on Habitat and Montezuma's Revenge.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 59 of its 76 references that are in the map. First posted on arXiv 5 Jun 2023.

Builds on

59 of 76 references.

References · 59

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  3. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  4. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,070
  5. 2018Exploration by Random Network Distillationcited by 993
  6. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  7. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  8. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  9. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 595
  10. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
  11. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  12. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 511
38 more
  1. 2018Is Q-learning Provably Efficient?cited by 487
  2. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 483
  3. 2020Planning to Explore via Self-Supervised World Modelscited by 368
  4. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 362
  5. 2018Quantifying Generalization in Reinforcement Learningcited by 352
  6. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 309
  7. 2018Large-Scale Study of Curiosity-Driven Learningcited by 280
  8. 2017Count-Based Exploration with Neural Density Modelscited by 257
  9. 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulationcited by 250
  10. 2015Contextual Markov Decision Processescited by 220
  11. 2020Never Give Up: Learning Directed Exploration Strategiescited by 212
  12. 2019Provably Efficient Exploration in Policy Optimizationcited by 205
  13. 2021Benchmarking the Spectrum of Agent Capabilitiescited by 200
  14. 2020The NetHack Learning Environmentcited by 198
  15. 2018A Study on Overfitting in Deep Reinforcement Learningcited by 174
  16. 2020iGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenescited by 155
  17. 2015Incentivizing Exploration In Reinforcement Learning With Deep Predictive Modelscited by 154
  18. 2020RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environmentscited by 135
  19. 2018Assessing Generalization in Deep Reinforcement Learningcited by 133
  20. 2018Generalization and Regularization in DQNcited by 118
  21. 2018Count-Based Exploration with the Successor Representationcited by 112
  22. 2018Model-Based Active Explorationcited by 105
  23. 2021MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Researchcited by 97
  24. 2020PC-PG: Policy Cover Directed Exploration for Provable Policy Gradient Learningcited by 92
  25. 2019Observational Overfitting in Reinforcement Learningcited by 91
  26. 2020Learning with AMIGo: Adversarially Motivated Intrinsic Goalscited by 88
  27. 2018A Dissection of Overfitting and Generalization in Continuous Reinforcement Learningcited by 87
  28. 2017Surprise-Based Intrinsic Motivation for Deep Reinforcement Learningcited by 83
  29. 2018Efficient Bias-Span-Constrained Exploration-Exploitation in Reinforcement Learningcited by 66
  30. 2019Obstacle Tower: A Generalization Challenge in Vision, Control, and Planningcited by 54
  31. 2022Semantic Exploration from Language Abstractions and Pretrained Representationscited by 51
  32. 2022Exploration via Elliptical Episodic Bonusescited by 51
  33. 2017Count-Based Exploration in Feature Space for Reinforcement Learningcited by 49
  34. 2022Improving Intrinsic Exploration with Language Abstractionscited by 49
  35. 2021Adversarially Guided Actor-Criticcited by 47
  36. 2021MADE: Exploration via Maximizing Deviation from Explored Regionscited by 40
  37. 2021Interesting Object, Curious Agent: Learning Task-Agnostic Explorationcited by 38
  38. 2019No-Regret Exploration in Goal-Oriented Reinforcement Learningcited by 34

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 6; 2025: 5; 2026: 5.

20 papers.

Sort:

By area:

  1. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 102
  2. 2024Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learningcited by 88
  3. 2023Maximum State Entropy Exploration using Predecessor and Successor Representationscited by 23
  4. 2025Episodic Novelty Through Temporal Distancecited by 16
  5. 2024RLeXplore: Accelerating Research in Intrinsically-Motivated Reinforcement Learningcited by 10
  6. 2025The impact of intrinsic rewards on exploration in Reinforcement Learningcited by 8
  7. 2023Improving Intrinsic Exploration by Creating Stationary Objectivescited by 7
  8. 2024Just Cluster It: An Approach for Exploration in High-Dimensions using Clustering and Pre-Trained Representationscited by 3
  9. 2025Beyond Noisy-TVs: Noise-Robust Exploration Via Learning Progress Monitoringcited by 2
  10. 2024Training on Irrelevant States Implies Data Augmentation: Generalization in Contextual MDPscited by 1
  11. 2025Deep Reinforcement Learning with Hybrid Intrinsic Reward Modelcited by 1
  12. 2025CAE: Repurposing the Critic as an Explorer in Deep Reinforcement Learningcited by 1
8 more
  1. 2026SLOPE: Optimistic Potential Landscape Shaping for Model-based Reinforcement Learningcited by 1
  2. 2026Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learningcited by 1
  3. 2023Enhanced Generalization through Prioritization and Diversity in Self-Imitation Reinforcement Learning over Procedural Environments with Sparse Rewardscited by 0
  4. 2024Explore-Go: Leveraging Exploration for Generalisation in Deep Reinforcement Learningcited by 0
  5. 2024Fostering Intrinsic Motivation in Reinforcement Learning with Pretrained Foundation Modelscited by 0
  6. 2026CIG: Exploration via Conditional Information Gaincited by 0
  7. 2026Breaking the Epistemic Trap: Active Perception Under Compound Uncertaintycited by 0
  8. 2026Reward Structure Shapes the Interaction Between Episodic Exploration and Neural Memory in Reinforcement Learningcited by 0