# Compatible Value Gradients for Reinforcement Learning of Continuous Deep Policies

arXiv: https://arxiv.org/abs/1509.03005 · HTML: https://mlsanity.com/paper/1509.03005/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Sep 2015.

## Built on by (most cited first, 5 of 5)

- [Solving Heterogeneous General Equilibrium Economic Models with Deep Reinforcement Learning](https://mlsanity.com/paper/2103.16977/) (2021)
- [Compatible Gradient Approximations for Actor-Critic Algorithms](https://mlsanity.com/paper/2409.01477/) (2024)
- [Empirical Bayesian Multi-Bandit Learning](https://mlsanity.com/paper/2510.26284/) (2025)
- [Taylor TD-learning](https://mlsanity.com/paper/2302.14182/) (2023)
- [Soft Deterministic Policy Gradient with Gaussian Smoothing](https://mlsanity.com/paper/2605.06228/) (2026)

## Abstract (arXiv)

This paper proposes GProp, a deep reinforcement learning algorithm for continuous policies with compatible function approximation. The algorithm is based on two innovations. Firstly, we present a temporal-difference based method for learning the gradient of the value-function. Secondly, we present the deviator-actor-critic (DAC) model, which comprises three neural networks that estimate the value function, its gradient, and determine the actor's policy respectively. We evaluate GProp on two challenging tasks: a contextual bandit problem constructed from nonparametric regression datasets that is designed to probe the ability of reinforcement learning algorithms to accurately estimate gradients; and the octopus arm, a challenging reinforcement learning benchmark. GProp is competitive with fully supervised methods on the bandit task and achieves the best performance to date on the octopus arm.