Reinforcement Learning from Demonstrations by Novel Interactive Expert and Application to Automatic Berthing Control Systems for Unmanned Surface Vessel
On the map: Offline RL and imitation learning from data → Imitation learning methods
Abstract (arXiv)
In this paper, two novel practical methods of Reinforcement Learning from Demonstration (RLfD) are developed and applied to automatic berthing control systems for Unmanned Surface Vessel. A new expert data generation method, called Model Predictive Based Expert (MPBE) which combines Model Predictive Control and Deep Deterministic Policy Gradient, is developed to provide high quality supervision data for RLfD algorithms. A straightforward RLfD method, model predictive Deep Deterministic Policy Gradient (MP-DDPG), is firstly introduced by replacing the RL agent with MPBE to directly interact with the environment. Then distribution mismatch problem is analyzed for MP-DDPG, and two techniques that alleviate distribution mismatch are proposed. Furthermore, another novel RLfD algorithm based on the MP-DDPG, called Self-Guided Actor-Critic (SGAC) is present, which can effectively leverage MPBE by continuously querying it to generate high quality expert data online. The distribution mismatch problem leading to unstable learning process is addressed by SGAC in a DAgger manner. In addition, theoretical analysis is given to prove that SGAC algorithm can converge with guaranteed monotonic improvement. Simulation results verify the effectiveness of MP-DDPG and SGAC to accomplish the ship berthing control task, and show advantages of SGAC comparing with other typical reinforcement learning algorithms and MP-DDPG.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 15 of its 48 references that are in the map. First posted on arXiv 23 Feb 2022.
Builds on
15 of 48 references.
Closest ancestors · 2
Same direction, newest first
- 2018Truncated Horizon Policy Search: Combining Reinforcement Learning & Imitation Learning29 May
- 2018Reinforcement Learning from Imperfect Demonstrations14 Feb
Other references · 13
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2015Continuous control with deep reinforcement learningcited by 3,279
- 2015Trust Region Policy Optimizationcited by 2,503
- 2021Convex optimizationcited by 2,479
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
- 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
- 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 288
- 2014Reinforcement and Imitation Learning via Interactive No-Regret Learningcited by 116
- 2019Exploring Model-based Planning with Policy Networkscited by 90
- 2015Learning Deep Control Policies for Autonomous Aerial Vehicles with MPC-Guided Policy Searchcited by 41
- 2015Learning Contact-Rich Manipulation Skills with Guided Policy Searchcited by 37
- 2019Reinforcement Learning from Imperfect Demonstrations under Soft Expert Guidancecited by 25
1 more
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2020Exploration-efficient Deep Reinforcement Learning with Demonstration Guidance for Robot Control
- 2022Accelerating Self-Imitation Learning from Demonstrations via Policy Constraints and Q-Ensemble
- 2024Towards an Adaptable and Generalizable Optimization Engine in Decision and Control: A Meta Reinforcement Learning Approach
- 2020Learning Sparse Rewarded Tasks from Sub-Optimal Demonstrations
- 2022A Survey on Model-based Reinforcement Learning
- 2021Efficiently Training On-Policy Actor-Critic Networks in Robotic Deep Reinforcement Learning with Demonstration-like Sampled Exploration
- 2024Learning Autonomous Docking Operation of Fully Actuated Autonomous Surface Vessel from Expert data
- 2024Is FISHER All You Need in The Multi-AUV Underwater Target Tracking Task?