# Sim-to-reality adaptation for Deep Reinforcement Learning applied to an underwater docking application

arXiv: https://arxiv.org/abs/2603.12020 · HTML: https://mlsanity.com/paper/2603.12020/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for robot navigation and UAV control, one of 15 directions in Deep reinforcement learning methods.
- Builds on 4 of its 18 references that are in the map.
- First posted on arXiv 12 Mar 2026.

## Placed in

- Deep reinforcement learning methods / [RL for robot navigation and UAV control](https://mlsanity.com/deeprl/rl-for-robot-navigation-and-uav-control/)

## Builds on (most cited first, 4 of 4)

- [Stonefish: Supporting Machine Learning Research in Marine Robotics](https://mlsanity.com/paper/2502.11887/) (2025)
- [Deep Reinforcement Learning for Continuous Docking Control of Autonomous Underwater Vehicles: A Benchmarking Study](https://mlsanity.com/paper/2108.02665/) (2021)
- [Deep Interactive Reinforcement Learning for Path Following of Autonomous Underwater Vehicle](https://mlsanity.com/paper/2001.03359/) (2020)
- [Fast Policy Learning for 6-DOF Position Control of Underwater Vehicles](https://mlsanity.com/paper/2512.13359/) (2025)

## Abstract (arXiv)

Deep Reinforcement Learning (DRL) offers a robust alternative to traditional control methods for autonomous underwater docking, particularly in adapting to unpredictable environmental conditions. However, bridging the "sim-to-real" gap and managing high training latencies remain significant bottlenecks for practical deployment. This paper presents a systematic approach for autonomous docking using the Girona Autonomous Underwater Vehicle (AUV) by leveraging a high-fidelity digital twin environment. We adapted the Stonefish simulator into a multiprocessing RL framework to significantly accelerate the learning process while incorporating realistic AUV dynamics, collision models, and sensor noise. Using the Proximal Policy Optimization (PPO) algorithm, we developed a 6-DoF control policy trained in a headless environment with randomized starting positions to ensure generalized performance. Our reward structure accounts for distance, orientation, action smoothness, and adaptive collision penalties to facilitate soft docking. Experimental results demonstrate that the agent achieved a success rate of over 90% in simulation. Furthermore, successful validation in a physical test tank confirmed the efficacy of the sim-to-reality adaptation, with the DRL controller exhibiting emergent behaviors such as pitch-based braking and yaw oscillations to assist in mechanical alignment.