# Depth Control of Model-Free AUVs via Reinforcement Learning

arXiv: https://arxiv.org/abs/1711.08224 · HTML: https://mlsanity.com/paper/1711.08224/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Nov 2017.

## Built on by (most cited first, 4 of 4)

- [Deep Reinforcement Learning for Continuous Docking Control of Autonomous Underwater Vehicles: A Benchmarking Study](https://mlsanity.com/paper/2108.02665/) (2021)
- [Decentralized Circle Formation Control for Fish-like Robots in the Real-world via Reinforcement Learning](https://mlsanity.com/paper/2103.05293/) (2021)
- [Feeding control and water quality monitoring in aquaculture systems: Opportunities and challenges](https://mlsanity.com/paper/2306.09920/) (2023)
- [Average-Power-Budgeted Underwater Vehicle Control via Constrained Reinforcement Learning](https://mlsanity.com/paper/2606.25680/) (2026)

## Abstract (arXiv)

In this paper, we consider depth control problems of an autonomous underwater vehicle (AUV) for tracking the desired depth trajectories. Due to the unknown dynamical model of the AUV, the problems cannot be solved by most of model-based controllers. To this purpose, we formulate the depth control problems of the AUV as continuous-state, continuous-action Markov decision processes (MDPs) under unknown transition probabilities. Based on deterministic policy gradient (DPG) and neural network approximation, we propose a model-free reinforcement learning (RL) algorithm that learns a state-feedback controller from sampled trajectories of the AUV. To improve the performance of the RL algorithm, we further propose a batch-learning scheme through replaying previous prioritized trajectories. We illustrate with simulations that our model-free method is even comparable to the model-based controllers as LQI and NMPC. Moreover, we validate the effectiveness of the proposed RL algorithm on a seafloor data set sampled from the South China Sea.