# Vessel-following model for inland waterways based on deep reinforcement learning

arXiv: https://arxiv.org/abs/2207.03257 · HTML: https://mlsanity.com/paper/2207.03257/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 71 references that are in the map.
- First posted on arXiv 7 Jul 2022.

## Builds on (most cited first, 17 of 17)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Reinforcement Learning for Robust Parameterized Locomotion Control of Bipedal Robots](https://mlsanity.com/paper/2103.14295/) (2021)
- [Imitating Driver Behavior with Generative Adversarial Networks](https://mlsanity.com/paper/1701.06699/) (2017)
- [Safe, Efficient, and Comfortable Velocity Control based on Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/1902.00089/) (2019)
- [Human-Like Autonomous Car-Following Model with Deep Reinforcement Learning](https://mlsanity.com/paper/1901.00569/) (2019)
- [Dissipation of stop-and-go waves via control of autonomous vehicles: Field experiments](https://mlsanity.com/paper/1705.01693/) (2017)
- [Comparison of Deep Reinforcement Learning and Model Predictive Control for Adaptive Cruise Control](https://mlsanity.com/paper/1910.12047/) (2019)
- [Controlling an Autonomous Vehicle with Deep Reinforcement Learning](https://mlsanity.com/paper/1909.12153/) (2019)
- [Longitudinal Dynamic versus Kinematic Models for Car-Following Control Using Deep Reinforcement Learning](https://mlsanity.com/paper/1905.08314/) (2019)
- [Modified DDPG car-following model with a real-world human driving experience with CARLA simulator](https://mlsanity.com/paper/2112.14602/) (2021)
- [Comparing Numerical Integration Schemes for Time-Continuous Car-Following Models](https://mlsanity.com/paper/1403.4881/) (2014)
- [Dampen the Stop-and-Go Traffic with Connected and Automated Vehicles -- A Deep Reinforcement Learning Approach](https://mlsanity.com/paper/2005.08245/) (2020)
- [Formulation and validation of a car-following model based on deep reinforcement learning](https://mlsanity.com/paper/2109.14268/) (2021)

## Built on by (most cited first, 2 of 2)

- [Spatial-temporal recurrent reinforcement learning for autonomous ships](https://mlsanity.com/paper/2211.01004/) (2022)
- [2-Level Reinforcement Learning for Ships on Inland Waterways: Path Planning and Following](https://mlsanity.com/paper/2307.16769/) (2023)

## Abstract (arXiv)

While deep reinforcement learning (RL) has been increasingly applied in designing car-following models in the last years, this study aims at investigating the feasibility of RL-based vehicle-following for complex vehicle dynamics and strong environmental disturbances. As a use case, we developed an inland waterways vessel-following model based on realistic vessel dynamics, which considers environmental influences, such as varying stream velocity and river profile. We extracted natural vessel behavior from anonymized AIS data to formulate a reward function that reflects a realistic driving style next to comfortable and safe navigation. Aiming at high generalization capabilities, we propose an RL training environment that uses stochastic processes to model leading trajectory and river dynamics. To validate the trained model, we defined different scenarios that have not been seen in training, including realistic vessel-following on the Middle Rhine. Our model demonstrated safe and comfortable driving in all scenarios, proving excellent generalization abilities. Furthermore, traffic oscillations could effectively be dampened by deploying the trained model on a sequence of following vessels.