mlsanityData to 7 Oct 2026

Policy Optimization with Smooth Guidance Learned from State-Only Demonstrations

arXiv 2401.00162 ↗First posted 30 Dec 2023cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

The sparsity of reward feedback remains a challenging problem in online deep reinforcement learning (DRL). Previous approaches have utilized offline demonstrations to achieve impressive results in multiple hard tasks. However, these approaches place high demands on demonstration quality, and obtaining expert-like actions is often costly and unrealistic. To tackle these problems, we propose a simple and efficient algorithm called Policy Optimization with Smooth Guidance (POSG), which leverages a small set of state-only demonstrations (where expert action information is not included in demonstrations) to indirectly make approximate and feasible long-term credit assignments and facilitate exploration. Specifically, we first design a trajectory-importance evaluation mechanism to determine the quality of the current trajectory against demonstrations. Then, we introduce a guidance reward computation technology based on trajectory importance to measure the impact of each state-action pair, fusing the demonstrator's state distribution with reward information into the guidance reward. We theoretically analyze the performance improvement caused by smooth guidance rewards and derive a new worst-case lower bound on the performance improvement. Extensive results demonstrate POSG's significant advantages in control performance and convergence speed in four sparse-reward environments, including the grid-world maze, Hopper-v4, HalfCheetah-v4, and Ant maze. Notably, the specific metrics and quantifiable results are investigated to demonstrate the superiority of POSG.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 50 of its 68 references that are in the map. First posted on arXiv 30 Dec 2023.

Builds on

50 of 68 references.

References · 50

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  6. 2016OpenAI Gymcited by 1,684
  7. 2018World Modelscited by 1,444
  8. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  9. 2017Constrained Policy Optimizationcited by 928
  10. 2019Model-Based Reinforcement Learning for Ataricited by 495
  11. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  12. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
38 more
  1. 2020First return, then explorecited by 256
  2. 2020Never Give Up: Learning Directed Exploration Strategiescited by 210
  3. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 188
  4. 2019Efficient Exploration via State Marginal Matchingcited by 186
  5. 2018Generative Adversarial Imitation from Observationcited by 183
  6. 2018RUDDER: Return Decomposition for Delayed Rewardscited by 162
  7. 2018Meta-Gradient Reinforcement Learningcited by 156
  8. 2017Neural Episodic Controlcited by 148
  9. 2018Self-Imitation Learningcited by 135
  10. 2017Stochastic Neural Networks for Hierarchical Reinforcement Learningcited by 124
  11. 2016Model-Free Episodic Controlcited by 100
  12. 2019Hindsight Credit Assignmentcited by 74
  13. 2019State Alignment-based Imitation Learningcited by 73
  14. 2020Counterfactual Credit Assignment in Model-Free Reinforcement Learningcited by 60
  15. 2018Optimizing Agent Behavior over Long Time Scales by Transporting Valuecited by 57
  16. 2021Hierarchical Reinforcement Learning By Discovering Intrinsic Optionscited by 57
  17. 2022Reinforcement Learning with Sparse Rewards using Guidance from Offline Demonstrationcited by 54
  18. 2019Making Efficient Use of Demonstrations to Solve Hard Exploration Problemscited by 45
  19. 2021MADE: Exploration via Maximizing Deviation from Explored Regionscited by 40
  20. 2018Truncated Horizon Policy Search: Combining Reinforcement Learning & Imitation Learningcited by 39
  21. 2020State-only Imitation with Transition Dynamics Mismatchcited by 37
  22. 2019Diversity-Inducing Policy Gradient: Using Maximum Mean Discrepancy to Find a Set of Diverse Policiescited by 34
  23. 2020Learning Guidance Rewards with Trajectory-space Smoothingcited by 30
  24. 2018Learning Self-Imitating Diverse Policiescited by 29
  25. 2018Sample-Efficient Deep Reinforcement Learning via Episodic Backward Updatecited by 29
  26. 2021Synthetic Returns for Long-Term Credit Assignmentcited by 27
  27. 2019Watch, Try, Learn: Meta-Learning from Demonstrations and Rewardcited by 26
  28. 2018Generative Adversarial Self-Imitation Learningcited by 25
  29. 2019Reinforcement Learning from Imperfect Demonstrations under Soft Expert Guidancecited by 25
  30. 2021Generalizable Episodic Memory for Deep Reinforcement Learningcited by 22
  31. 2020Guided Exploration with Proximal Policy Optimization using a Single Demonstrationcited by 19
  32. 2021Active Hierarchical Exploration with Stable Subgoal Representation Learningcited by 17
  33. 2017Pre-training Neural Networks with Human Demonstrations for Deep Reinforcement Learningcited by 15
  34. 2021Model-Based Episodic Memory Induces Dynamic Hybrid Controlscited by 14
  35. 2022Imitation Learning from Observations under Transition Model Disparitycited by 11
  36. 2022Rewarding Episodic Visitation Discrepancy for Exploration in Reinforcement Learningcited by 7
  37. 2019Self-Attentional Credit Assignment for Transfer in Reinforcement Learningcited by 2
  38. 2021Adaptive Pairwise Weights for Temporal Credit Assignmentcited by 2

Built on by

Not cited yet.