# Zeroth-order Deterministic Policy Gradient

arXiv: https://arxiv.org/abs/2006.07314 · HTML: https://mlsanity.com/paper/2006.07314/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jun 2020.

## Built on by (most cited first, 9 of 9)

- [Learning Optimal Deterministic Policies with Stochastic Policy Gradients](https://mlsanity.com/paper/2405.02235/) (2024)
- [Memory-Efficient Gradient Unrolling for Large-Scale Bi-level Optimization](https://mlsanity.com/paper/2406.14095/) (2024)
- [Examining average and discounted reward optimality criteria in reinforcement learning](https://mlsanity.com/paper/2107.01348/) (2021)
- [Distributed Multi-Agent Reinforcement Learning Based on Graph-Induced Local Value Functions](https://mlsanity.com/paper/2202.13046/) (2022)
- [Distributed Cooperative Multi-Agent Reinforcement Learning with Directed Coordination Graph](https://mlsanity.com/paper/2201.04962/) (2022)
- [A nearly Blackwell-optimal policy gradient method](https://mlsanity.com/paper/2105.13609/) (2021)
- [Model-Free Learning of Optimal Deterministic Resource Allocations in Wireless Systems via Action-Space Exploration](https://mlsanity.com/paper/2108.10352/) (2021)
- [Deterministic Policy Gradient Primal-Dual Methods for Continuous-Space Constrained MDPs](https://mlsanity.com/paper/2408.10015/) (2024)
- [Soft Deterministic Policy Gradient with Gaussian Smoothing](https://mlsanity.com/paper/2605.06228/) (2026)

## Abstract (arXiv)

Deterministic Policy Gradient (DPG) removes a level of randomness from standard randomized-action Policy Gradient (PG), and demonstrates substantial empirical success for tackling complex dynamic problems involving Markov decision processes. At the same time, though, DPG loses its ability to learn in a model-free (i.e., actor-only) fashion, frequently necessitating the use of critics in order to obtain consistent estimates of the associated policy-reward gradient. In this work, we introduce Zeroth-order Deterministic Policy Gradient (ZDPG), which approximates policy-reward gradients via two-point stochastic evaluations of the Q-function, constructed by properly designed low-dimensional action-space perturbations. Exploiting the idea of random horizon rollouts for obtaining unbiased estimates of the Q-function, ZDPG lifts the dependence on critics and restores true model-free policy learning, while enjoying built-in and provable algorithmic stability. Additionally, we present new finite sample complexity bounds for ZDPG, which improve upon existing results by up to two orders of magnitude. Our findings are supported by several numerical experiments, which showcase the effectiveness of ZDPG in a practical setting, and its advantages over both PG and Baseline PG.