# Navigation in a simplified Urban Flow through Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2409.17922 · HTML: https://mlsanity.com/paper/2409.17922/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot navigation and UAV control, one of 15 directions in Deep reinforcement learning methods.
- Builds on 20 of its 69 references that are in the map.
- First posted on arXiv 26 Sep 2024.

## Placed in

- Deep reinforcement learning methods / [RL for robot navigation and UAV control](https://mlsanity.com/deeprl/rl-for-robot-navigation-and-uav-control/)

## Builds on (most cited first, 20 of 20)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations](https://mlsanity.com/paper/2003.08938/) (2020)
- [Enhancing Computational Fluid Dynamics with Machine Learning](https://mlsanity.com/paper/2110.02085/) (2021)
- [PEORL: Integrating Symbolic Planning and Hierarchical Reinforcement Learning for Robust Decision-Making](https://mlsanity.com/paper/1804.07779/) (2018)
- [A Review on Deep Learning in UAV Remote Sensing](https://mlsanity.com/paper/2101.10861/) (2021)
- [Effective control of two-dimensional Rayleigh--B'enard convection: invariant multi-agent reinforcement learning is all you need](https://mlsanity.com/paper/2304.02370/) (2023)
- [Memory-based Deep Reinforcement Learning for Obstacle Avoidance in UAV with Limited Environment Knowledge](https://mlsanity.com/paper/1811.03307/) (2018)
- [Learning Efficient Navigation in Vortical Flow Fields](https://mlsanity.com/paper/2102.10536/) (2021)
- [Zermelo's problem: Optimal point-to-point navigation in 2D turbulent flows using Reinforcement Learning](https://mlsanity.com/paper/1907.08591/) (2019)
- [A Vision Based Deep Reinforcement Learning Algorithm for UAV Obstacle Avoidance](https://mlsanity.com/paper/2103.06403/) (2021)
- [Emerging trends in machine learning for computational fluid dynamics](https://mlsanity.com/paper/2211.15145/) (2022)
- [Dynamic Feature-based Deep Reinforcement Learning for Flow Control of Circular Cylinder with Sparse Surface Pressure Sensing](https://mlsanity.com/paper/2307.01995/) (2023)
- [Effects of a Hovering Unmanned Aerial Vehicle on Urban Soundscapes Perception](https://mlsanity.com/paper/1912.00087/) (2019)
- [Active flow control for three-dimensional cylinders through deep reinforcement learning](https://mlsanity.com/paper/2309.02462/) (2023)
- [Importance of Environment Design in Reinforcement Learning: A Study of a Robotic Environment](https://mlsanity.com/paper/2102.10447/) (2021)
- [An Integrated Real-time UAV Trajectory Optimization with Potential Field Approach for Dynamic Collision Avoidance](https://mlsanity.com/paper/2303.02043/) (2023)

## Built on by (most cited first, 5 of 5)

- [Transformer-Based Fault-Tolerant Control for Fixed-Wing UAVs Using Knowledge Distillation and In-Context Adaptation](https://mlsanity.com/paper/2411.02975/) (2024)
- [Latent feedback control of distributed systems in multiple scenarios through deep learning-based reduced order models](https://mlsanity.com/paper/2412.09942/) (2024)
- [Navigation in a Three-Dimensional Urban Flow using Deep Reinforcement Learning](https://mlsanity.com/paper/2510.25679/) (2025)
- [Timescale Separation Enables Deep Reinforcement Learning Control of Rotating Detonation Engine Mode Transitions](https://mlsanity.com/paper/2604.14398/) (2026)
- [Explainable deep reinforcement learning reveals energy-efficient control strategies for turbulent drag reduction](https://mlsanity.com/paper/2606.00949/) (2026)

## Abstract (arXiv)

The increasing number of unmanned aerial vehicles (UAVs) in urban environments requires a strategy to minimize their environmental impact, both in terms of energy efficiency and noise reduction. In order to reduce these concerns, novel strategies for developing prediction models and optimization of flight planning, for instance through deep reinforcement learning (DRL), are needed. Our goal is to develop DRL algorithms capable of enabling the autonomous navigation of UAVs in urban environments, taking into account the presence of buildings and other UAVs, optimizing the trajectories in order to reduce both energetic consumption and noise. This is achieved using fluid-flow simulations which represent the environment in which UAVs navigate and training the UAV as an agent interacting with an urban environment. In this work, we consider a domain domain represented by a two-dimensional flow field with obstacles, ideally representing buildings, extracted from a three-dimensional high-fidelity numerical simulation. The presented methodology, using PPO+LSTM cells, was validated by reproducing a simple but fundamental problem in navigation, namely the Zermelo's problem, which deals with a vessel navigating in a turbulent flow, travelling from a starting point to a target location, optimizing the trajectory. The current method shows a significant improvement with respect to both a simple PPO and a TD3 algorithm, with a success rate (SR) of the PPO+LSTM trained policy of 98.7%, and a crash rate (CR) of 0.1%, outperforming both PPO (SR = 75.6%, CR=18.6%) and TD3 (SR=77.4% and CR=14.5%). This is the first step towards DRL strategies which will guide UAVs in a three-dimensional flow field using real-time signals, making the navigation efficient in terms of flight time and avoiding damages to the vehicle.