Sample-efficient Real-time Planning with Curiosity Cross-Entropy Method and Contrastive Learning
Not on the map. Why?
Abstract (arXiv)
Model-based reinforcement learning (MBRL) with real-time planning has shown great potential in locomotion and manipulation control tasks. However, the existing planning methods, such as the Cross-Entropy Method (CEM), do not scale well to complex high-dimensional environments. One of the key reasons for underperformance is the lack of exploration, as these planning methods only aim to maximize the cumulative extrinsic reward over the planning horizon. Furthermore, planning inside the compact latent space in the absence of observations makes it challenging to use curiosity-based intrinsic motivation. We propose Curiosity CEM (CCEM), an improved version of the CEM algorithm for encouraging exploration via curiosity. Our proposed method maximizes the sum of state-action Q values over the planning horizon, in which these Q values estimate the future extrinsic and intrinsic reward, hence encouraging reaching novel observations. In addition, our model uses contrastive representation learning to efficiently learn latent representations. Experiments on image-based continuous control tasks from the DeepMind Control suite show that CCEM is by a large margin more sample-efficient than previous MBRL algorithms and compares favorably with the best model-free RL methods.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 37 of its 47 references that are in the map. First posted on arXiv 7 Mar 2023.
Builds on
37 of 47 references.
References · 37
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
- 2016Layer Normalizationcited by 5,579
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
- 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
- 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
- 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
- 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
- 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
- 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
- 2022Temporal Difference Learning for Model Predictive Controlcited by 375
25 more
- 2020dm_control: Software and Tasks for Continuous Controlcited by 368
- 2020Planning to Explore via Self-Supervised World Modelscited by 365
- 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
- 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
- 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 288
- 2018Large-Scale Study of Curiosity-Driven Learningcited by 277
- 2019Self-Supervised Exploration via Disagreementcited by 263
- 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
- 2017Count-Based Exploration with Neural Density Modelscited by 255
- 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 152
- 2019Unsupervised State Representation Learning in Ataricited by 144
- 2015Variational Information Maximisation for Intrinsically Motivated Reinforcement Learningcited by 142
- 2020Learning to Utilize Shaping Rewards: A New Approach of Reward Shapingcited by 112
- 2019Exploring Model-based Planning with Policy Networkscited by 90
- 2021DreamerPro: Reconstruction-Free Model-Based Reinforcement Learning with Prototypical Representationscited by 73
- 2020Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstructioncited by 66
- 2020Predictive Information Accelerates Learning in RLcited by 56
- 2021Temporal Predictive Coding For Model-Based Planning In Latent Spacecited by 54
- 2021Sample Efficient Reinforcement Learning via Model-Ensemble Exploration and Exploitationcited by 17
- 2020Latent World Models For Intrinsically Motivated Explorationcited by 15
- 2021Sample-efficient Reinforcement Learning Representation Learning with Curiosity Contrastive Forward Dynamics Modelcited by 14
- 2023Internally Rewarded Reinforcement Learningcited by 14
- 2022Integrating Contrastive Learning with Dynamic Models for Reinforcement Learning from Imagescited by 12
- 2018Deep Intrinsically Motivated Continuous Actor-Critic for Efficient Robotic Visuomotor Skill Learningcited by 5
- 2022Value Summation: A Novel Scoring Function for MPC-based Model-based Reinforcement Learningcited by 1
Built on by
Citing papers per year: 2023: 1; 2024: 1; 2025: 1.
3 papers.