mlsanityData to 14 Oct 2026

Improving Intrinsic Exploration by Creating Stationary Objectives

arXiv 2310.18144 ↗First posted 27 Oct 2023cs.LG cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

Exploration bonuses in reinforcement learning guide long-horizon exploration by defining custom intrinsic objectives. Several exploration objectives like count-based bonuses, pseudo-counts, and state-entropy maximization are non-stationary and hence are difficult to optimize for the agent. While this issue is generally known, it is usually omitted and solutions remain under-explored. The key contribution of our work lies in transforming the original non-stationary rewards into stationary rewards through an augmented state representation. For this purpose, we introduce the Stationary Objectives For Exploration (SOFE) framework. SOFE requires identifying sufficient statistics for different exploration bonuses and finding an efficient encoding of these statistics to use as input to a deep network. SOFE is based on proposing state augmentations that expand the state space but hold the promise of simplifying the optimization of the agent's objective. We show that SOFE improves the performance of several exploration objectives, including count-based bonuses, pseudo-counts, and state-entropy maximization. Moreover, SOFE outperforms prior methods that attempt to stabilize the optimization of intrinsic objectives. We demonstrate the efficacy of SOFE in hard-exploration problems, including sparse-reward tasks, pixel-based observations, 3D navigation, and procedurally generated environments.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 46 of its 58 references that are in the map. First posted on arXiv 27 Oct 2023.

Builds on

46 of 58 references.

References · 46

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  7. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  8. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  9. 2018Exploration by Random Network Distillationcited by 993
  10. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  11. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  12. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
34 more
  1. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
  2. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 567
  3. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 511
  4. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 483
  5. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 309
  6. 2017Noisy Networks for Explorationcited by 266
  7. 2017Count-Based Exploration with Neural Density Modelscited by 257
  8. 2019Efficient Exploration via State Marginal Matchingcited by 188
  9. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 139
  10. 2020RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environmentscited by 135
  11. 2019Behaviour Suite for Reinforcement Learningcited by 132
  12. 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 122
  13. 2018Count-Based Exploration with the Successor Representationcited by 112
  14. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 108
  15. 2023A Tutorial on Meta-Reinforcement Learningcited by 108
  16. 2021MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Researchcited by 97
  17. 2019Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learningcited by 93
  18. 2020Reinforcement Learning for Non-Stationary Markov Decision Processes: The Blessing of (More) Optimismcited by 76
  19. 2022Exploration via Elliptical Episodic Bonusescited by 51
  20. 2021Adversarially Guided Actor-Criticcited by 47
  21. 2019Non-Stationary Markov Decision Processes, a Worst-Case Approach using Model-Based Reinforcement Learning, Extended versioncited by 36
  22. 2020Griddly: A platform for AI research in gamescited by 31
  23. 2023Flipping Coins to Estimate Pseudocounts for Exploration in Reinforcement Learningcited by 28
  24. 2020Deep Reinforcement Learning for Navigation in AAA Video Gamescited by 27
  25. 2021Decoupled Reinforcement Learning to Stabilise Intrinsically-Motivated Explorationcited by 25
  26. 2023Maximum State Entropy Exploration using Predecessor and Successor Representationscited by 23
  27. 2023Explore to Generalize in Zero-Shot RLcited by 20
  28. 2023A Study of Global and Episodic Bonuses for Exploration in Contextual MDPscited by 20
  29. 2018On Evaluating the Generalization of LSTM Models in Formal Languagescited by 17
  30. 2021Decoupled Exploration and Exploitation Policies for Sample-Efficient Reinforcement Learningcited by 14
  31. 2020Quantifying the Impact of Non-Stationarity in Reinforcement Learning-Based Traffic Signal Controlcited by 6
  32. 2021Graph augmented Deep Reinforcement Learning in the GameRLand3D environmentcited by 5
  33. 2023Approximate information state based convergence analysis of recurrent Q-learningcited by 5
  34. 2021Godot Reinforcement Learning Agentscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 2; 2026: 2.

7 papers.

  1. 2024RLeXplore: Accelerating Research in Intrinsically-Motivated Reinforcement Learningcited by 10
  2. 2024Surprise-Adaptive Intrinsic Motivation for Unsupervised Reinforcement Learningcited by 4
  3. 2025Topology Enhanced MARL for Multi-Agent Cooperative Decision-Making of CAVscited by 1
  4. 2023Augmenting Unsupervised Reinforcement Learning with Self-Referencecited by 0
  5. 2025KEA: Keeping Exploration Alive by Proactively Coordinating Exploration Strategiescited by 0
  6. 2026Remember to be Curious: Episodic Context and Persistent Worlds for 3D Explorationcited by 0
  7. 2026Principled Direction-Free Intrinsic Motivation through Model-Free Epistemic Free-Energy Estimatorscited by 0