# Deep Reinforcement Learning-based UAV Navigation and Control: A Soft Actor-Critic with Hindsight Experience Replay Approach

arXiv: https://arxiv.org/abs/2106.01016 · HTML: https://mlsanity.com/paper/2106.01016/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 40 references that are in the map.
- First posted on arXiv 2 Jun 2021.

## Builds on (most cited first, 14 of 14)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Reach-Avoid Problems with Time-Varying Dynamics, Targets and Constraints](https://mlsanity.com/paper/1410.6445/) (2014)
- [Deep Reinforcement Learning based Automatic Exploration for Navigation in Unknown Environment](https://mlsanity.com/paper/2007.11808/) (2020)
- [The problem with DDPG: understanding failures in deterministic environments with sparse rewards](https://mlsanity.com/paper/1911.11679/) (2019)
- [Parameter Optimization and Learning in a Spiking Neural Network for UAV Obstacle Avoidance targeting Neuromorphic Processors](https://mlsanity.com/paper/1910.07960/) (2019)
- [Decentralized Motion Planning with Collision Avoidance for a Team of UAVs under High Level Goals](https://mlsanity.com/paper/1610.00912/) (2016)

## Built on by (most cited first, 4 of 4)

- [When Maximum Entropy Misleads Policy Optimization](https://mlsanity.com/paper/2506.05615/) (2025)
- [VeRLPy: Python Library for Verification of Digital Designs with Reinforcement Learning](https://mlsanity.com/paper/2108.03978/) (2021)
- [Vision-Based Deep Reinforcement Learning of UAV Autonomous Navigation Using Privileged Information](https://mlsanity.com/paper/2412.06313/) (2024)
- [Reinforcement Learning for Autonomous Point-to-Point UAV Navigation](https://mlsanity.com/paper/2509.13943/) (2025)

## Abstract (arXiv)

In this paper, we propose SACHER (soft actor-critic (SAC) with hindsight experience replay (HER)), which constitutes a class of deep reinforcement learning (DRL) algorithms. SAC is known as an off-policy model-free DRL algorithm based on the maximum entropy framework, which outperforms earlier DRL algorithms in terms of exploration, robustness and learning performance. However, in SAC, maximizing the entropy-augmented objective may degrade the optimality of learning outcomes. HER is known as a sample-efficient replay method that enhances the performance of off-policy DRL algorithms by allowing the agent to learn from both failures and successes. We apply HER to SAC and propose SACHER to improve the learning performance of SAC. More precisely, SACHER achieves the desired optimal outcomes faster and more accurately than SAC, since HER improves the sample efficiency of SAC. We apply SACHER to the navigation and control problem of unmanned aerial vehicles (UAVs), where SACHER generates the optimal navigation path of the UAV under various obstacles in operation. Specifically, we show the effectiveness of SACHER in terms of the tracking error and cumulative reward in UAV operation by comparing them with those of state-of-the-art DRL algorithms, SAC and DDPG. Note that SACHER in UAV navigation and control problems can be applied to arbitrary models of UAVs.