AFU: Actor-Free critic Updates in off-policy RL for continuous control
Not on the map. Why?
Abstract (arXiv)
This paper presents AFU, an off-policy deep RL algorithm addressing in a new way the challenging "max-Q problem" in Q-learning for continuous action spaces, with a solution based on regression and conditional gradient scaling. AFU has an actor but its critic updates are entirely independent from it. As a consequence, the actor can be chosen freely. In the initial version, AFU-alpha, we employ the same stochastic actor as in Soft Actor-Critic (SAC), but we then study a simple failure mode of SAC and show how AFU can be modified to make actor updates less likely to become trapped in local optima, resulting in a second version of the algorithm, AFU-beta. Experimental results demonstrate the sample efficiency of both versions of AFU, marking it as the first model-free off-policy algorithm competitive with state-of-the-art actor-critic methods while departing from the actor-critic perspective.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 22 of its 37 references that are in the map. First posted on arXiv 24 Apr 2024.
Builds on
22 of 37 references.
References · 22
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2015Continuous control with deep reinforcement learningcited by 3,279
- 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
- 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
- 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
- 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
- 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
- 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
- 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
- 2016Input Convex Neural Networkscited by 407
- 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
10 more
- 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
- 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 154
- 2016Continuous Deep Q-Learning with Model-based Accelerationcited by 149
- 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 114
- 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 113
- 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 108
- 2017Discrete Sequential Prediction of Continuous Actions for Deep RLcited by 50
- 2019CAQL: Continuous Action Q-Learningcited by 22
- 2023Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Criticcited by 20
- 2021Aggressive Q-Learning with Ensembles: Achieving Both High Sample Efficiency and High Asymptotic Performancecited by 7
Built on by
Citing papers per year: 2026: 1.
1 paper.