Compatible Gradient Approximations for Actor-Critic Algorithms
Not on the map. Why?
Abstract (arXiv)
Deterministic policy gradient algorithms are foundational for actor-critic methods in controlling continuous systems, yet they often encounter inaccuracies due to their dependence on the derivative of the critic's value estimates with respect to input actions. This reliance requires precise action-value gradient computations, a task that proves challenging under function approximation. We introduce an actor-critic algorithm that bypasses the need for such precision by employing a zeroth-order approximation of the action-value gradient through two-point stochastic gradient estimation within the action space. This approach provably and effectively addresses compatibility issues inherent in deterministic policy gradient schemes. Empirical results further demonstrate that our algorithm not only matches but frequently exceeds the performance of current state-of-the-art methods by a substantial extent.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 20 of its 38 references that are in the map. First posted on arXiv 2 Sep 2024.
Builds on
20 of 38 references.
References · 20
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2015Continuous control with deep reinforcement learningcited by 3,279
- 2015Trust Region Policy Optimizationcited by 2,503
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
- 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
- 2016OpenAI Gymcited by 1,684
- 2017Deep Reinforcement Learning that Matterscited by 727
- 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 555
- 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
- 2013Optimal rates for zero-order convex optimization: the power of two function evaluationscited by 190
- 2018Policy Optimization via Importance Samplingcited by 47
8 more
- 2019Neural Empirical Bayescited by 46
- 2021Local policy search with Bayesian optimizationcited by 37
- 2020Parameter-Based Value Functionscited by 24
- 2019On approximating nabla f with neural networkscited by 18
- 2020How to Learn a Useful Critic? Model-based Action-Gradient-Estimator Policy Optimizationcited by 16
- 2015Compatible Value Gradients for Reinforcement Learning of Continuous Deep Policiescited by 5
- 2022Deep Intrinsically Motivated Exploration in Continuous Controlcited by 4
- 2022Mitigating Off-Policy Bias in Actor-Critic Methods with One-Step Q-learning: A Novel Correction Approachcited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.