mlsanityData to 7 Oct 2026

Generative Planning for Temporally Coordinated Exploration in Reinforcement Learning

arXiv 2201.09765 ↗First posted 24 Jan 2022cs.LG cs.AI cs.CVcited by 10

Not on the map. Why?

Abstract (arXiv)

Standard model-free reinforcement learning algorithms optimize a policy that generates the action to be taken in the current time step in order to maximize expected future return. While flexible, it faces difficulties arising from the inefficient exploration due to its single step nature. In this work, we present Generative Planning method (GPM), which can generate actions not only for the current step, but also for a number of future steps (thus termed as generative planning). This brings several benefits to GPM. Firstly, since GPM is trained by maximizing value, the plans generated from it can be regarded as intentional action sequences for reaching high value regions. GPM can therefore leverage its generated multi-step plans for temporally coordinated exploration towards high value regions, which is potentially more effective than a sequence of actions generated by perturbing each action at single step level, whose consistent movement decays exponentially with the number of exploration steps. Secondly, starting from a crude initial plan generator, GPM can refine it to be adaptive to the task, which, in return, benefits future explorations. This is potentially more effective than commonly used action-repeat strategy, which is non-adaptive in its form of plans. Additionally, since the multi-step plan can be interpreted as the intent of the agent from now to a span of time period into the future, it offers a more informative and intuitive signal for interpretation. Experiments are conducted on several benchmark environments and the results demonstrated its effectiveness compared with several baseline methods.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 49 of its 66 references that are in the map. First posted on arXiv 24 Jan 2022.

Builds on

49 of 66 references.

References · 49

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  6. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  7. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  8. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  9. 2016End to End Learning for Self-Driving Carscited by 1,000
  10. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  11. 2016The Option-Critic Architecturecited by 433
  12. 2020Planning to Explore via Self-Supervised World Modelscited by 365
37 more
  1. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  2. 2017Noisy Networks for Explorationcited by 262
  3. 2019CoverNet: Multimodal Behavior Prediction using Trajectory Setscited by 259
  4. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
  5. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 220
  6. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 188
  7. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 171
  8. 2017Parameter Space Noise for Explorationcited by 158
  9. 2019End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordancescited by 147
  10. 2017Imagination-Augmented Agents for Deep Reinforcement Learningcited by 137
  11. 2017Stochastic Neural Networks for Hierarchical Reinforcement Learningcited by 124
  12. 2015Gradient Estimation Using Stochastic Computation Graphscited by 120
  13. 2017Value Prediction Networkcited by 117
  14. 2018Model-Based Active Explorationcited by 105
  15. 2018Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Controlcited by 104
  16. 2020Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learningcited by 100
  17. 2016The Predictron: End-To-End Learning and Planningcited by 92
  18. 2019Exploring Model-based Planning with Policy Networkscited by 90
  19. 2018Deep Imitative Models for Flexible Inference, Planning, and Controlcited by 75
  20. 2019Exploration via Hindsight Goal Generationcited by 53
  21. 2018Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddingscited by 52
  22. 2020Continuous-Discrete Reinforcement Learning for Hybrid Control in Roboticscited by 42
  23. 2020Behavior Priors for Efficient Reinforcement Learningcited by 40
  24. 2017Learning to Repeat: Fine Grained Action Repetition for Deep Reinforcement Learningcited by 38
  25. 2016Watch This: Scalable Cost-Function Learning for Path Planning in Urban Environmentscited by 33
  26. 2019Driving with Style: Inverse Reinforcement Learning in General-Purpose Planning for Automated Drivingcited by 30
  27. 2016Strategic Attentive Writer for Learning Macro-Actionscited by 29
  28. 2019The Natural Language of Actionscited by 28
  29. 2021TempoRL: Learning When to Actcited by 25
  30. 2019Explicit Explore-Exploit Algorithms in Continuous State Spacescited by 19
  31. 2019Learning to Predict Without Looking Ahead: World Models Without Forward Predictioncited by 15
  32. 2021TAAC: Temporally Abstract Actor-Critic for Continuous Controlcited by 14
  33. 2019Autoregressive Policies for Continuous Control Deep Reinforcement Learningcited by 13
  34. 2020Locally Persistent Exploration in Continuous Control Tasks with Sparse Rewardscited by 12
  35. 2018Model Learning for Look-ahead Exploration in Continuous Controlcited by 4
  36. 2018NADPEx: An on-policy temporally consistent exploration method for deep reinforcement learningcited by 3
  37. 2019Regularizing Neural Networks for Future Trajectory Prediction via Inverse Reinforcement Learning Frameworkcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 2; 2024: 3; 2025: 2; 2026: 1.

10 papers.

  1. 2023Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learningcited by 114
  2. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 103
  3. 2024FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learningcited by 26
  4. 2025Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returnscited by 16
  5. 2024Knowledge Planning in Large Language Models for Domain-Aligned Counseling Summarizationcited by 11
  6. 2026RAD-2: Scaling Reinforcement Learning in a Generator-Discriminator Frameworkcited by 7
  7. 2022Model-based Reinforcement Learning with Multi-step Plan Value Estimationcited by 6
  8. 2022Leveraging Demonstrations with Latent Space Priorscited by 2
  9. 2024Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Controlcited by 0
  10. 2025Select before Act: Spatially Decoupled Action Repetition for Continuous Controlcited by 0