Learning Off-policy with Model-based Intrinsic Motivation For Active Online Exploration

arXiv 2404.00651 ↗First posted 31 Mar 2024cs.LG cs.AIcited by 1

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Recent advancements in deep reinforcement learning (RL) have demonstrated notable progress in sample efficiency, spanning both model-based and model-free paradigms. Despite the identification and mitigation of specific bottlenecks in prior works, the agent's exploration ability remains under-emphasized in the realm of sample-efficient RL. This paper investigates how to achieve sample-efficient exploration in continuous control tasks. We introduce an RL algorithm that incorporates a predictive model and off-policy learning elements, where an online planner enhanced by a novelty-aware terminal value function is employed for sample collection. Leveraging the forward predictive error within a latent state space, we derive an intrinsic reward without incurring parameters overhead. This reward establishes a solid connection to model uncertainty, allowing the agent to effectively overcome the asymptotic performance gap. Through extensive experiments, our method shows competitive or even superior performance compared to prior works, especially the sparse reward cases.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 50 of its 57 references that are in the map. First posted on arXiv 31 Mar 2024.

Builds on

50 of 57 references.

References · 50

2016Layer Normalizationcited by 5,669
2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
2017Hindsight Experience Replaycited by 1,136

38 more

2018Exploration by Random Network Distillationcited by 993
2018DeepMind Control Suitecited by 956
2020Object-Centric Learning with Slot Attentioncited by 876
2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
2022Temporal Difference Learning for Model Predictive Controlcited by 397
2023Efficient Online Reinforcement Learning with Offline Datacited by 363
2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
2018Large-Scale Study of Curiosity-Driven Learningcited by 280
2022The Primacy Bias in Deep Reinforcement Learningcited by 247
2021Mastering Atari Games with Limited Datacited by 237
2019Contrastive Learning of Structured World Modelscited by 209
2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 168
2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 162
2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 156
2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 114
2018Model-Based Active Explorationcited by 105
2022Deep Hierarchical Planning from Pixelscited by 101
2020Sample-efficient Cross-Entropy Method for Real-time Planningcited by 95
2022A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learningcited by 83
2022BYOL-Explore: Exploration by Bootstrapped Predictioncited by 73
2022Denoised MDPs: Learning World Models Better Than the World Itselfcited by 68
2017Data-efficient Deep Reinforcement Learning for Dexterous Manipulationcited by 63
2020Learning Off-Policy with Online Planningcited by 53
2021The Difficulty of Passive Learning in Deep Reinforcement Learningcited by 50
2023Efficient Deep Reinforcement Learning Requires Regulating Overfittingcited by 36
2023Facing Off World Model Backbones: RNNs, Transformers, and S4cited by 31
2023A Generalist Dynamics Model for Controlcited by 24
2023Reparameterized Policy Learning for Multimodal Trajectory Optimizationcited by 20
2021Collect & Infer -- a fresh look at data-efficient Reinforcement Learningcited by 17
2023Simplified Temporal Consistency Reinforcement Learningcited by 17
2021Evaluating model-based planning and planner amortization for continuous controlcited by 15
2023IQL-TD-MPC: Implicit Q-Learning for Hierarchical Model Predictive Controlcited by 14
2023Efficient RL via Disentangled Environment and Agent Representationscited by 12
2022Active Exploration for Robotic Manipulationcited by 10
2023Efficient Sparse-Reward Goal-Conditioned Reinforcement Learning with a High Replay Ratio and Regularizationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

2026Decoupling Exploration and Policy Optimization: Uncertainty Guided Tree Search for Hard Explorationcited by 1

Similar, not cited

Close in wording, no citation either way