mlsanityData to 7 Oct 2026

Decoupled Reinforcement Learning to Stabilise Intrinsically-Motivated Exploration

arXiv 2107.08966 ↗First posted 19 Jul 2021cs.LG cs.AIcited by 25

Not on the map. Why?

Abstract (arXiv)

Intrinsic rewards can improve exploration in reinforcement learning, but the exploration process may suffer from instability caused by non-stationary reward shaping and strong dependency on hyperparameters. In this work, we introduce Decoupled RL (DeRL) as a general framework which trains separate policies for intrinsically-motivated exploration and exploitation. Such decoupling allows DeRL to leverage the benefits of intrinsic rewards for exploration while demonstrating improved robustness and sample efficiency. We evaluate DeRL algorithms in two sparse-reward environments with multiple types of intrinsic rewards. Our results show that DeRL is more robust to varying scale and rate of decay of intrinsic rewards and converges to the same evaluation returns than intrinsically-motivated baselines in fewer interactions. Lastly, we discuss the challenge of distribution shift and show that divergence constraint regularisers can successfully minimise instability caused by divergence of exploration and exploitation policies.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 26 of its 44 references that are in the map. First posted on arXiv 19 Jul 2021.

Builds on

26 of 44 references.

References · 26

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  7. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  8. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  9. 2018Exploration by Random Network Distillationcited by 966
  10. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  11. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  12. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
14 more
  1. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  2. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  3. 2018Large-Scale Study of Curiosity-Driven Learningcited by 277
  4. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  5. 2017Count-Based Exploration with Neural Density Modelscited by 255
  6. 2020Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Taskscited by 207
  7. 2020RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environmentscited by 135
  8. 2019Behaviour Suite for Reinforcement Learningcited by 129
  9. 2020Shared Experience Actor-Critic for Multi-Agent Reinforcement Learningcited by 114
  10. 2020Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrificescited by 59
  11. 2021Adversarially Guided Actor-Criticcited by 47
  12. 2020Optimistic Exploration even with a Pessimistic Initialisationcited by 25
  13. 2019Benchmarking Bonus-Based Exploration Methods on the Arcade Learning Environmentcited by 17
  14. 2021Decoupled Exploration and Exploitation Policies for Sample-Efficient Reinforcement Learningcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 3; 2023: 13; 2024: 2; 2025: 4; 2026: 2.

25 papers.

Sort:

By area:

  1. 2023Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learningcited by 45
  2. 2023Conditional Mutual Information for Disentangled Representations in Reinforcement Learningcited by 27
  3. 2022Redeeming Intrinsic Rewards via Constrained Optimizationcited by 24
  4. 2023Accelerating Reinforcement Learning with Value-Conditional State Entropy Explorationcited by 22
  5. 2021Robust On-Policy Sampling for Data-Efficient Policy Evaluation in Reinforcement Learningcited by 15
  6. 2023Intrinsic Language-Guided Exploration for Complex Long-Horizon Robotic Manipulation Taskscited by 8
  7. 2022Multi-Horizon Representations with Hierarchical Forward Models for Reinforcement Learningcited by 7
  8. 2023Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Biascited by 7
  9. 2023Improving Intrinsic Exploration by Creating Stationary Objectivescited by 7
  10. 2023On the Theory of Risk-Aware Agents: Bridging Actor-Critic and Economicscited by 6
  11. 2023Ensemble Value Functions for Efficient Exploration in Multi-Agent Reinforcement Learningcited by 5
  12. 2023Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learningcited by 4
13 more
  1. 2022Deep Reinforcement Learning for Multi-Agent Interactioncited by 3
  2. 2023Efficient Reinforcement Learning via Decoupling Exploration and Utilizationcited by 3
  3. 2024ETGL-DDPG: A Deep Deterministic Policy Gradient Algorithm for Sparse Reward Continuous Controlcited by 2
  4. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  5. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environmentscited by 1
  6. 2025NashPG: A Policy Gradient Method with Iteratively Refined Regularization for Finding Nash Equilibriacited by 1
  7. 2026Fairness over Equality: Correcting Social Incentives in Asymmetric Sequential Social Dilemmascited by 1
  8. 2023DIRECT: Learning from Sparse and Shifting Rewards using Discriminative Reward Co-Trainingcited by 0
  9. 2023Augmenting Unsupervised Reinforcement Learning with Self-Referencecited by 0
  10. 2024MESA: Cooperative Meta-Exploration in Multi-Agent Learning through Exploiting State-Action Space Structurecited by 0
  11. 2025Optimistic ε-Greedy Exploration for Cooperative Multi-Agent Reinforcement Learningcited by 0
  12. 2025Redistributing Rewards Across Time and Agents for Multi-Agent Reinforcement Learningcited by 0
  13. 2025KEA: Keeping Exploration Alive by Proactively Coordinating Exploration Strategiescited by 0