# Aquatic Navigation: A Challenging Benchmark for Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2405.20534 · HTML: https://mlsanity.com/paper/2405.20534/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot navigation and UAV control, one of 15 directions in Deep reinforcement learning methods.
- Builds on 32 of its 53 references that are in the map.
- First posted on arXiv 30 May 2024.

## Placed in

- Deep reinforcement learning methods / [RL for robot navigation and UAV control](https://mlsanity.com/deeprl/rl-for-robot-navigation-and-uav-control/)

## Builds on (most cited first, 30 of 32)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Safe Reinforcement Learning via Shielding](https://mlsanity.com/paper/1708.08611/) (2017)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05143/) (2016)
- [Responsive Safety in Reinforcement Learning by PID Lagrangian Methods](https://mlsanity.com/paper/2007.03964/) (2020)
- [Unity: A General Platform for Intelligent Agents](https://mlsanity.com/paper/1809.02627/) (2018)
- [Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO](https://mlsanity.com/paper/2005.12729/) (2020)
- [Learning to Utilize Shaping Rewards: A New Approach of Reward Shaping](https://mlsanity.com/paper/2011.02669/) (2020)
- [Virtual to Real Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/1704.03952/) (2017)
- [Towards Scalable Verification of Deep Reinforcement Learning](https://mlsanity.com/paper/2105.11931/) (2021)
- [Verifying Learning-Based Robotic Navigation Systems](https://mlsanity.com/paper/2205.13536/) (2022)
- [Safe Reinforcement Learning using Formal Verification for Tissue Retraction in Autonomous Robotic-Assisted Surgery](https://mlsanity.com/paper/2109.02323/) (2021)
- [Formally Explaining Neural Networks within Reactive Systems](https://mlsanity.com/paper/2308.00143/) (2023)
- [Embodied Visual Navigation with Automatic Curriculum Learning in Real Environments](https://mlsanity.com/paper/2009.05429/) (2020)
- [The #DNN-Verification Problem: Counting Unsafe Inputs for Deep Neural Networks](https://mlsanity.com/paper/2301.07068/) (2023)
- [Centralizing State-Values in Dueling Networks for Multi-Robot Reinforcement Learning Mapless Navigation](https://mlsanity.com/paper/2112.09012/) (2021)
- [Reducing Variance in Temporal-Difference Value Estimation via Ensemble of Deep Networks](https://mlsanity.com/paper/2209.07670/) (2022)
- [Reinforcement Learning from Delayed Observations via World Models](https://mlsanity.com/paper/2403.12309/) (2024)
- [Verifying Generalization in Deep Learning](https://mlsanity.com/paper/2302.05745/) (2023)
- [Benchmarking Safe Deep Reinforcement Learning in Aquatic Navigation](https://mlsanity.com/paper/2112.10593/) (2021)
- [Scenario-Assisted Deep Reinforcement Learning](https://mlsanity.com/paper/2202.04337/) (2022)

## Built on by (most cited first, 1 of 1)

- [RoboRAN: A Unified Robotics Framework for Reinforcement Learning-Based Autonomous Navigation](https://mlsanity.com/paper/2505.14526/) (2025)

## Abstract (arXiv)

An exciting and promising frontier for Deep Reinforcement Learning (DRL) is its application to real-world robotic systems. While modern DRL approaches achieved remarkable successes in many robotic scenarios (including mobile robotics, surgical assistance, and autonomous driving) unpredictable and non-stationary environments can pose critical challenges to such methods. These features can significantly undermine fundamental requirements for a successful training process, such as the Markovian properties of the transition model. To address this challenge, we propose a new benchmarking environment for aquatic navigation using recent advances in the integration between game engines and DRL. In more detail, we show that our benchmarking environment is problematic even for state-of-the-art DRL approaches that may struggle to generate reliable policies in terms of generalization power and safety. Specifically, we focus on PPO, one of the most widely accepted algorithms, and we propose advanced training techniques (such as curriculum learning and learnable hyperparameters). Our extensive empirical evaluation shows that a well-designed combination of these ingredients can achieve promising results. Our simulation environment and training baselines are freely available to facilitate further research on this open problem and encourage collaboration in the field.