# Asynchronous Episodic Deep Deterministic Policy Gradient: Towards Continuous Control in Computationally Complex Environments

arXiv: https://arxiv.org/abs/1903.00827 · HTML: https://mlsanity.com/paper/1903.00827/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Mar 2019.

## Built on by (most cited first, 14 of 14)

- [CACTO: Continuous Actor-Critic with Trajectory Optimization -- Towards global optimality](https://mlsanity.com/paper/2211.06625/) (2022)
- [SelectAugment: Hierarchical Deterministic Sample Selection for Data Augmentation](https://mlsanity.com/paper/2112.02862/) (2021)
- [Solving Continuous Control with Episodic Memory](https://mlsanity.com/paper/2106.08832/) (2021)
- [Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples with On-Policy Experience](https://mlsanity.com/paper/2109.11767/) (2021)
- [Neural Episodic Control with State Abstraction](https://mlsanity.com/paper/2301.11490/) (2023)
- [Asynchronous Curriculum Experience Replay: A Deep Reinforcement Learning Approach for UAV Autonomous Motion Control in Unknown Dynamic Environments](https://mlsanity.com/paper/2207.01251/) (2022)
- [LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models](https://mlsanity.com/paper/2505.15293/) (2025)
- [Computational Performance of Deep Reinforcement Learning to find Nash Equilibria](https://mlsanity.com/paper/2104.12895/) (2021)
- [Double Actor-Critic with TD Error-Driven Regularization in Reinforcement Learning](https://mlsanity.com/paper/2409.19231/) (2024)
- [Auxiliary Task-based Deep Reinforcement Learning for Quantum Control](https://mlsanity.com/paper/2302.14312/) (2023)
- [Robust Stability of Neural Network-controlled Nonlinear Systems with Parametric Variability](https://mlsanity.com/paper/2109.05710/) (2021)
- [RangL: A Reinforcement Learning Competition Platform](https://mlsanity.com/paper/2208.00003/) (2022)
- [Bag of Policies for Distributional Deep Exploration](https://mlsanity.com/paper/2308.01759/) (2023)
- [Learning From Failures: Efficient Reinforcement Learning Control with Episodic Memory](https://mlsanity.com/paper/2603.07110/) (2026)

## Abstract (arXiv)

Deep Deterministic Policy Gradient (DDPG) has been proved to be a successful reinforcement learning (RL) algorithm for continuous control tasks. However, DDPG still suffers from data insufficiency and training inefficiency, especially in computationally complex environments. In this paper, we propose Asynchronous Episodic DDPG (AE-DDPG), as an expansion of DDPG, which can achieve more effective learning with less training time required. First, we design a modified scheme for data collection in an asynchronous fashion. Generally, for asynchronous RL algorithms, sample efficiency or/and training stability diminish as the degree of parallelism increases. We consider this problem from the perspectives of both data generation and data utilization. In detail, we re-design experience replay by introducing the idea of episodic control so that the agent can latch on good trajectories rapidly. In addition, we also inject a new type of noise in action space to enrich the exploration behaviors. Experiments demonstrate that our AE-DDPG achieves higher rewards and requires less time consuming than most popular RL algorithms in Learning to Run task which has a computationally complex environment. Not limited to the control tasks in computationally complex environments, AE-DDPG also achieves higher rewards and 2- to 4-fold improvement in sample efficiency on average compared to other variants of DDPG in MuJoCo environments. Furthermore, we verify the effectiveness of each proposed technique component through abundant ablation study.