# Spatial-temporal recurrent reinforcement learning for autonomous ships

arXiv: https://arxiv.org/abs/2211.01004 · HTML: https://mlsanity.com/paper/2211.01004/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 130 references that are in the map.
- First posted on arXiv 2 Nov 2022.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [Optimization of Molecules via Deep Reinforcement Learning](https://mlsanity.com/paper/1810.08678/) (2018)
- [Motion Planning Among Dynamic, Decision-Making Agents with Deep Reinforcement Learning](https://mlsanity.com/paper/1805.01956/) (2018)
- [Memory-based control with recurrent neural networks](https://mlsanity.com/paper/1512.04455/) (2015)
- [Memory-based Deep Reinforcement Learning for POMDPs](https://mlsanity.com/paper/2102.12344/) (2021)
- [Collision Avoidance in Pedestrian-Rich Environments with Deep Reinforcement Learning](https://mlsanity.com/paper/1910.11689/) (2019)
- [Taming an autonomous surface vehicle for path following and collision avoidance using deep reinforcement learning](https://mlsanity.com/paper/1912.08578/) (2019)
- [Comparing Numerical Integration Schemes for Time-Continuous Car-Following Models](https://mlsanity.com/paper/1403.4881/) (2014)
- [Risk-based implementation of COLREGs for autonomous surface vehicles using deep reinforcement learning](https://mlsanity.com/paper/2112.00115/) (2021)
- [Roboat II: A Novel Autonomous Surface Vessel for Urban Environments](https://mlsanity.com/paper/2007.10220/) (2020)
- [Addressing Maximization Bias in Reinforcement Learning with Two-Sample Testing](https://mlsanity.com/paper/2201.08078/) (2022)
- [Hybrid Collision Avoidance for ASVs Compliant with COLREGs Rules 8 and 13-17](https://mlsanity.com/paper/1907.00198/) (2019)
- [COLREGs-Informed RRT* for Collision Avoidance of Marine Crafts](https://mlsanity.com/paper/2103.14426/) (2021)
- [Vessel-following model for inland waterways based on deep reinforcement learning](https://mlsanity.com/paper/2207.03257/) (2022)

## Built on by (most cited first, 3 of 3)

- [2-Level Reinforcement Learning for Ships on Inland Waterways: Path Planning and Following](https://mlsanity.com/paper/2307.16769/) (2023)
- [Self-organized free-flight arrival for urban air mobility](https://mlsanity.com/paper/2404.03710/) (2024)
- [Two-step dynamic obstacle avoidance](https://mlsanity.com/paper/2311.16841/) (2023)

## Abstract (arXiv)

This paper proposes a spatial-temporal recurrent neural network architecture for deep Q-networks that can be used to steer an autonomous ship. The network design makes it possible to handle an arbitrary number of surrounding target ships while offering robustness to partial observability. Furthermore, a state-of-the-art collision risk metric is proposed to enable an easier assessment of different situations by the agent. The COLREG rules of maritime traffic are explicitly considered in the design of the reward function. The final policy is validated on a custom set of newly created single-ship encounters called `Around the Clock' problems and the commonly used Imazu (1987) problems, which include 18 multi-ship scenarios. Performance comparisons with artificial potential field and velocity obstacle methods demonstrate the potential of the proposed approach for maritime path planning. Furthermore, the new architecture exhibits robustness when it is deployed in multi-agent scenarios and it is compatible with other deep reinforcement learning algorithms, including actor-critic frameworks.