# Learning to Swim: Reinforcement Learning for 6-DOF Control of Thruster-driven Autonomous Underwater Vehicles

arXiv: https://arxiv.org/abs/2410.00120 · HTML: https://mlsanity.com/paper/2410.00120/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot navigation and UAV control, one of 15 directions in Deep reinforcement learning methods.
- Builds on 21 of its 38 references that are in the map.
- First posted on arXiv 30 Sep 2024.

## Placed in

- Deep reinforcement learning methods / [RL for robot navigation and UAV control](https://mlsanity.com/deeprl/rl-for-robot-navigation-and-uav-control/)

## Builds on (most cited first, 21 of 21)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning](https://mlsanity.com/paper/2109.11978/) (2021)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [DayDreamer: World Models for Physical Robot Learning](https://mlsanity.com/paper/2206.14176/) (2022)
- [Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments](https://mlsanity.com/paper/2301.04195/) (2023)
- [Learning Agile Robotic Locomotion Skills by Imitating Animals](https://mlsanity.com/paper/2004.00784/) (2020)
- [Rapid Locomotion via Reinforcement Learning](https://mlsanity.com/paper/2205.02824/) (2022)
- [Learning High-Speed Flight in the Wild](https://mlsanity.com/paper/2110.05113/) (2021)
- [Deep Drone Racing: From Simulation to Reality with Domain Randomization](https://mlsanity.com/paper/1905.09727/) (2019)
- [Sim-to-(Multi)-Real: Transfer of Low-Level Robust Control Policies to Multiple Quadrotors](https://mlsanity.com/paper/1903.04628/) (2019)
- [Learning Vision-Based Bipedal Locomotion for Challenging Terrain](https://mlsanity.com/paper/2309.14594/) (2023)
- [Learning to Fly in Seconds](https://mlsanity.com/paper/2311.13081/) (2023)
- [UNav-Sim: A Visually Realistic Underwater Robotics Simulator and Synthetic Data-generation Framework](https://mlsanity.com/paper/2310.11927/) (2023)
- [CUREE: A Curious Underwater Robot for Ecosystem Exploration](https://mlsanity.com/paper/2303.03943/) (2023)
- [DAVE Aquatic Virtual Environment: Toward a General Underwater Robotics Simulator](https://mlsanity.com/paper/2209.02862/) (2022)
- [From Concept to Field Tests: Accelerated Development of Multi-AUV Missions Using a High-Fidelity Faster-than-Real-Time Simulator](https://mlsanity.com/paper/2311.10377/) (2023)

## Built on by (most cited first, 6 of 6)

- [Never too Prim to Swim: An LLM-Enhanced RL-based Adaptive S-Surface Controller for AUVs under Extreme Sea Conditions](https://mlsanity.com/paper/2503.00527/) (2025)
- [Fast Policy Learning for 6-DOF Position Control of Underwater Vehicles](https://mlsanity.com/paper/2512.13359/) (2025)
- [Make Your AUV Adaptive: An Environment-Aware Reinforcement Learning Framework For Underwater Tasks](https://mlsanity.com/paper/2506.15082/) (2025)
- [Learning to Dock: A Simulation-based Study on Closing the Sim2Real Gap in Autonomous Underwater Docking](https://mlsanity.com/paper/2506.17823/) (2025)
- [Towards End to End Motion Planning and Execution for Autonomous Underwater Vehicles Using Reinforcement Learning](https://mlsanity.com/paper/2606.08513/) (2026)
- [AquaOrbit: Sim-to-Real Reinforcement Learning for Underwater Target Orbiting under Intermittent Visual Feedback](https://mlsanity.com/paper/2609.24054/) (2026)

## Abstract (arXiv)

Controlling AUVs can be challenging because of the effect of complex non-linear hydrodynamic forces acting on the robot, which are significant in water and cannot be ignored. The problem is exacerbated for small AUVs for which the dynamics can change significantly with payload changes and deployments under different hydrodynamic conditions. The common approach to AUV control is a combination of passive stabilization with added buoyancy on top and weights on the bottom, and a PID controller tuned for simple and smooth motion primitives. However, the approach comes at the cost of sluggish controls and often the need to re-tune controllers with configuration changes. In this paper, we propose a fast (trainable in minutes), reinforcement learning-based approach for full 6 degree of freedom (DOF) control of a thruster-driven AUVs, taking 6-DOF command-conditioned inputs direct to thruster outputs. We present a new, highly parallelized simulator for underwater vehicle dynamics. We demonstrate this approach through zero-shot sim-to-real (with no tuning) transfer onto a real AUV that produces comparable results to hand-tuned PID controllers. Furthermore, we show that domain randomization on the simulator produces policies that are robust to small variations in vehicle's physical parameters.