# Asynchronous Curriculum Experience Replay: A Deep Reinforcement Learning Approach for UAV Autonomous Motion Control in Unknown Dynamic Environments

arXiv: https://arxiv.org/abs/2207.01251 · HTML: https://mlsanity.com/paper/2207.01251/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 63 references that are in the map.
- First posted on arXiv 4 Jul 2022.

## Builds on (most cited first, 18 of 18)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Revisiting Fundamentals of Experience Replay](https://mlsanity.com/paper/2007.06700/) (2020)
- [A Deeper Look at Experience Replay](https://mlsanity.com/paper/1712.01275/) (2017)
- [Simultaneous Navigation and Radio Mapping for Cellular-Connected UAV with Deep Reinforcement Learning](https://mlsanity.com/paper/2003.07574/) (2020)
- [Memory-based Deep Reinforcement Learning for Obstacle Avoidance in UAV with Limited Environment Knowledge](https://mlsanity.com/paper/1811.03307/) (2018)
- [StarCraft Micromanagement with Reinforcement Learning and Curriculum Transfer Learning](https://mlsanity.com/paper/1804.00810/) (2018)
- [Path Planning for UAV-Mounted Mobile Edge Computing with Deep Reinforcement Learning](https://mlsanity.com/paper/2001.10268/) (2020)
- [Prioritized Sequence Experience Replay](https://mlsanity.com/paper/1905.12726/) (2019)
- [Deep Reinforcement Learning with Quantum-inspired Experience Replay](https://mlsanity.com/paper/2101.02034/) (2021)
- [Deep Reinforcement Learning Based Dynamic Trajectory Control for UAV-assisted Mobile Edge Computing](https://mlsanity.com/paper/1911.03887/) (2019)
- [Asynchronous Episodic Deep Deterministic Policy Gradient: Towards Continuous Control in Computationally Complex Environments](https://mlsanity.com/paper/1903.00827/) (2019)
- [Curriculum Learning of Visual Attribute Clusters for Multi-Task Classification](https://mlsanity.com/paper/1709.06664/) (2017)
- [Information Freshness-Aware Task Offloading in Air-Ground Integrated Edge Computing Systems](https://mlsanity.com/paper/2007.10129/) (2020)
- [Predictive PER: Balancing Priority and Diversity towards Stable Deep Reinforcement Learning](https://mlsanity.com/paper/2011.13093/) (2020)

## Built on by (most cited first, 4 of 4)

- [Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Survey](https://mlsanity.com/paper/2411.05614/) (2024)
- [Pre-trained Transformer-Enabled Strategies with Human-Guided Fine-Tuning for End-to-end Navigation of Autonomous Vehicles](https://mlsanity.com/paper/2402.12666/) (2024)
- [Analyzing Generalization in Policy Networks: A Case Study with the Double-Integrator System](https://mlsanity.com/paper/2312.10472/) (2023)
- [UAV Trajectory Optimization via Improved Noisy Deep Q-Network](https://mlsanity.com/paper/2602.05644/) (2026)

## Abstract (arXiv)

Unmanned aerial vehicles (UAVs) have been widely used in military warfare. In this paper, we formulate the autonomous motion control (AMC) problem as a Markov decision process (MDP) and propose an advanced deep reinforcement learning (DRL) method that allows UAVs to execute complex tasks in large-scale dynamic three-dimensional (3D) environments. To overcome the limitations of the prioritized experience replay (PER) algorithm and improve performance, the proposed asynchronous curriculum experience replay (ACER) uses multithreads to asynchronously update the priorities, assigns the true priorities and applies a temporary experience pool to make available experiences of higher quality for learning. A first-in-useless-out (FIUO) experience pool is also introduced to ensure the higher use value of the stored experiences. In addition, combined with curriculum learning (CL), a more reasonable training paradigm of sampling experiences from simple to difficult is designed for training UAVs. By training in a complex unknown environment constructed based on the parameters of a real UAV, the proposed ACER improves the convergence speed by 24.66% and the convergence result by 5.59% compared to the state-of-the-art twin delayed deep deterministic policy gradient (TD3) algorithm. The testing experiments carried out in environments with different complexities demonstrate the strong robustness and generalization ability of the ACER agent.