mlsanityData to 14 Oct 2026

Addressing Function Approximation Error in Actor-Critic Methods

arXiv 1802.09477 ↗First posted 26 Feb 2018cs.AI cs.LG stat.MLcited by 2,417

Not on the map. Why?

Cited by 825 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In value-based reinforcement learning methods such as deep Q-learning, function approximation errors are known to lead to overestimated value estimates and suboptimal policies. We show that this problem persists in an actor-critic setting and propose novel mechanisms to minimize its effects on both the actor and the critic. Our algorithm builds on Double Q-learning, by taking the minimum value between a pair of critics to limit overestimation. We draw the connection between target networks and overestimation bias, and suggest delaying policy updates to reduce per-update error and further improve performance. We evaluate our method on the suite of OpenAI gym tasks, outperforming the state of the art in every environment tested.

Key facts. Cited by 2,417 papers in the map (data through 14 Oct 2026). First posted on arXiv 26 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2018: 1; 2019: 7; 2020: 17; 2021: 330; 2022: 381; 2023: 403; 2024: 435; 2025: 453; 2026: 390.

2,417 papers.

Sort:

By area:

  1. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  2. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  3. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  4. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  5. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  6. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  7. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 516
  8. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 508
  9. 2022Temporal Difference Learning for Model Predictive Controlcited by 397
  10. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  11. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 370
  12. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
38 more
  1. 2022Behavior Transformers: Cloning k modes with one stonecited by 304
  2. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 302
  3. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  4. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  5. 2025Learning to Reason under Off-Policy Guidancecited by 281
  6. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  7. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 237
  8. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 231
  9. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 214
  10. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 213
  11. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learningcited by 204
  12. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 197
  13. 2021End-to-End Urban Driving by Imitating a Reinforcement Learning Coachcited by 194
  14. 2025Flow Q-Learningcited by 184
  15. 2021Reinforcement Learning with Prototypical Representationscited by 179
  16. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  17. 2021The Values Encoded in Machine Learning Researchcited by 172
  18. 2023Efficient Diffusion Policies for Offline Reinforcement Learningcited by 157
  19. 2021Offline RL Without Off-Policy Evaluationcited by 151
  20. 2022Legged Locomotion in Challenging Terrains using Egocentric Visioncited by 151
  21. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learningcited by 149
  22. 2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 146
  23. 2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 143
  24. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 138
  25. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matchingcited by 138
  26. 2024SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learningcited by 136
  27. 2021Robust Reinforcement Learning on State Observations with Learned Optimal Adversarycited by 135
  28. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 134
  29. 2020PLAS: Latent Action Space for Offline Reinforcement Learningcited by 133
  30. 2022Imitation Is Not Enough: Robustifying Imitation with Reinforcement Learning for Challenging Driving Scenarioscited by 129
  31. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 126
  32. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 126
  33. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  34. 2021Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisationcited by 122
  35. 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 122
  36. 2020Primal Wasserstein Imitation Learningcited by 121
  37. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  38. 2022Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learningcited by 119

50 most cited of 2,417 All 2,417 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
  2. 2026Revisiting Overestimation Bias Problem of Q-learning: Settling Large Discrete Action Space via Action Intersection
  3. 2025Actor-Critics Can Achieve Optimal Sample Efficiency
  4. 2026Second-Order Actor-Critic Methods for Discounted MDPs via Policy Hessian Decomposition
  5. 2025A Theoretical Justification for Asymmetric Actor-Critic Algorithms
  6. 2023Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learning
  7. 2022Beyond the Policy Gradient Theorem for Efficient Policy Updates in Actor-Critic Algorithms
  8. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies