mlsanityData to 7 Oct 2026

Learning Pessimism for Robust and Efficient Off-Policy Reinforcement Learning

arXiv 2110.03375 ↗First posted 7 Oct 2021cs.LG cs.AIcited by 18

Not on the map. Why?

Abstract (arXiv)

Off-policy deep reinforcement learning algorithms commonly compensate for overestimation bias during temporal-difference learning by utilizing pessimistic estimates of the expected target returns. In this work, we propose Generalized Pessimism Learning (GPL), a strategy employing a novel learnable penalty to enact such pessimism. In particular, we propose to learn this penalty alongside the critic with dual TD-learning, a new procedure to estimate and minimize the magnitude of the target returns bias with trivial computational cost. GPL enables us to accurately counteract overestimation bias throughout training without incurring the downsides of overly pessimistic targets. By integrating GPL with popular off-policy algorithms, we achieve state-of-the-art results in both competitive proprioceptive and pixel-based benchmarks.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 42 of its 67 references that are in the map. First posted on arXiv 7 Oct 2021.

Builds on

42 of 67 references.

References · 42

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2016Layer Normalizationcited by 5,579
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2021Convex optimizationcited by 2,479
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  8. 2016OpenAI Gymcited by 1,684
  9. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  10. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  11. 2018DeepMind Control Suitecited by 935
  12. 2020On Layer Normalization in the Transformer Architecturecited by 906
30 more
  1. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  2. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  3. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  4. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  5. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  6. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  7. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  8. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  9. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  10. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  11. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  12. 2021Accounting for Variance in Machine Learning Benchmarkscited by 195
  13. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 154
  14. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
  15. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
  16. 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
  17. 2017Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Controlcited by 92
  18. 2019Exploring Model-based Planning with Policy Networkscited by 90
  19. 2019Better Exploration with Optimistic Actor-Criticcited by 88
  20. 2020Discovering Reinforcement Learning Algorithmscited by 75
  21. 2020DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correctioncited by 71
  22. 2018Benchmarking Reinforcement Learning Algorithms on Real-World Robotscited by 64
  23. 2019Meta-Learning via Learned Losscited by 60
  24. 2021Tactical Optimism and Pessimism for Deep Reinforcement Learningcited by 55
  25. 2020Meta-Gradient Reinforcement Learning with an Objective Discovered Onlinecited by 52
  26. 2020On the model-based stochastic value gradient for continuous reinforcement learningcited by 46
  27. 2014On the folded normal distributioncited by 31
  28. 2020Softmax Deep Double Deterministic Policy Gradientscited by 31
  29. 2021Efficient Continuous Control with Double Actors and Regularized Criticscited by 20
  30. 2021Towards Deeper Deep Reinforcement Learning with Spectral Normalizationcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 3; 2023: 2; 2024: 5; 2025: 4; 2026: 3.

18 papers.

Sort:

By area:

  1. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 104
  2. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learningcited by 50
  3. 2022Stabilizing Off-Policy Deep Reinforcement Learning from Pixelscited by 32
  4. 2023Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Criticcited by 20
  5. 2023On the Theory of Risk-Aware Agents: Bridging Actor-Critic and Economicscited by 6
  6. 2021Adaptively Calibrated Critic Estimates for Deep Reinforcement Learningcited by 5
  7. 2024Simple Ingredients for Offline Reinforcement Learningcited by 4
  8. 2022An Experimental Comparison Between Temporal Difference and Residual Gradient with Neural Network Approximationcited by 2
  9. 2022Policy Gradient With Serial Markov Chain Reasoningcited by 2
  10. 2024Double Actor-Critic with TD Error-Driven Regularization in Reinforcement Learningcited by 2
  11. 2025On the Effect of Regularization in Policy Mirror Descentcited by 1
  12. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theorycited by 1
6 more
  1. 2024A Case for Validation Buffer in Pessimistic Actor-Criticcited by 0
  2. 2025Multi-Actor Multi-Critic Deep Deterministic Reinforcement Learning with a Novel Q-Ensemble Methodcited by 0
  3. 2025Mitigating Estimation Bias with Representation Learning in TD Error-Driven Regularizationcited by 0
  4. 2026Stochastic Actor-Critic: Mitigating Overestimation via Temporal Aleatoric Uncertaintycited by 0
  5. 2026A Pontryagin Method of Model-based Reinforcement Learning via Hamiltonian Actor-Criticcited by 0
  6. 2026Collaborative Weighting with Pessimistic Critic for Mitigating Overestimation in Off-Policy Reinforcement Learningcited by 0