# 2-Level Reinforcement Learning for Ships on Inland Waterways: Path Planning and Following

arXiv: https://arxiv.org/abs/2307.16769 · HTML: https://mlsanity.com/paper/2307.16769/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 103 references that are in the map.
- First posted on arXiv 25 Jul 2023.

## Builds on (most cited first, 13 of 13)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned](https://mlsanity.com/paper/2102.02915/) (2021)
- [Memory-based Deep Reinforcement Learning for POMDPs](https://mlsanity.com/paper/2102.12344/) (2021)
- [COLREG-Compliant Collision Avoidance for Unmanned Surface Vehicle using Deep Reinforcement Learning](https://mlsanity.com/paper/2006.09540/) (2020)
- [Comparing Numerical Integration Schemes for Time-Continuous Car-Following Models](https://mlsanity.com/paper/1403.4881/) (2014)
- [Risk-based implementation of COLREGs for autonomous surface vehicles using deep reinforcement learning](https://mlsanity.com/paper/2112.00115/) (2021)
- [Enhanced method for reinforcement learning based dynamic obstacle avoidance by assessment of collision risk](https://mlsanity.com/paper/2212.04123/) (2022)
- [Addressing Maximization Bias in Reinforcement Learning with Two-Sample Testing](https://mlsanity.com/paper/2201.08078/) (2022)
- [Spatial-temporal recurrent reinforcement learning for autonomous ships](https://mlsanity.com/paper/2211.01004/) (2022)
- [Robust Path Following on Rivers Using Bootstrapped Reinforcement Learning](https://mlsanity.com/paper/2303.15178/) (2023)
- [Vessel-following model for inland waterways based on deep reinforcement learning](https://mlsanity.com/paper/2207.03257/) (2022)

## Built on by (most cited first, 2 of 2)

- [Self-organized free-flight arrival for urban air mobility](https://mlsanity.com/paper/2404.03710/) (2024)
- [Two-step dynamic obstacle avoidance](https://mlsanity.com/paper/2311.16841/) (2023)

## Abstract (arXiv)

This paper proposes a realistic modularized framework for controlling autonomous surface vehicles (ASVs) on inland waterways (IWs) based on deep reinforcement learning (DRL). The framework improves operational safety and comprises two levels: a high-level local path planning (LPP) unit and a low-level path following (PF) unit, each consisting of a DRL agent. The LPP agent is responsible for planning a path under consideration of dynamic vessels, closing a gap in the current research landscape. In addition, the LPP agent adequately considers traffic rules and the geometry of the waterway. We thereby introduce a novel application of a spatial-temporal recurrent neural network architecture to continuous action spaces. The LPP agent outperforms a state-of-the-art artificial potential field (APF) method by increasing the minimum distance to other vessels by 65% on average. The PF agent performs low-level actuator control while accounting for shallow water influences and the environmental forces winds, waves, and currents. Compared with a proportional-integral-derivative (PID) controller, the PF agent yields only 61% of the mean cross-track error (MCTE) while significantly reducing control effort (CE) in terms of the required absolute rudder angle. Lastly, both agents are jointly validated in simulation, employing the lower Elbe in northern Germany as an example case and using real automatic identification system (AIS) trajectories to model the behavior of other ships.