mlsanityData to 7 Oct 2026

Tactical Optimism and Pessimism for Deep Reinforcement Learning

arXiv 2102.03765 ↗First posted 7 Feb 2021cs.LGcited by 55

Not on the map. Why?

Abstract (arXiv)

In recent years, deep off-policy actor-critic algorithms have become a dominant approach to reinforcement learning for continuous control. One of the primary drivers of this improved performance is the use of pessimistic value updates to address function approximation errors, which previously led to disappointing performance. However, a direct consequence of pessimism is reduced exploration, running counter to theoretical support for the efficacy of optimism in the face of uncertainty. So which approach is best? In this work, we show that the most effective degree of optimism can vary both across tasks and over the course of learning. Inspired by this insight, we introduce a novel deep actor-critic framework, Tactical Optimistic and Pessimistic (TOP) estimation, which switches between optimistic and pessimistic value learning online. This is achieved by formulating the selection as a multi-arm bandit problem. We show in a series of continuous control tasks that TOP outperforms existing methods which rely on a fixed degree of optimism, setting a new state of the art in challenging pixel-based environments. Since our changes are simple to implement, we believe these insights can easily be incorporated into a multitude of off-policy algorithms.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Builds on 40 of its 73 references that are in the map. First posted on arXiv 7 Feb 2021.

Builds on

40 of 73 references.

References · 40

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2016OpenAI Gymcited by 1,684
  5. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  6. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  7. 2018DeepMind Control Suitecited by 935
  8. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  9. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  10. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  11. 2016Deep Exploration via Bootstrapped DQNcited by 518
  12. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
28 more
  1. 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
  2. 2018Is Q-learning Provably Efficient?cited by 485
  3. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  4. 2020Reinforcement Learning with Augmented Datacited by 452
  5. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  6. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  7. 2019Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Boundcited by 199
  8. 2020Effective Diversity in Population Based Reinforcement Learningcited by 116
  9. 2016Corralling a Band of Bandit Algorithmscited by 99
  10. 2019Better Exploration with Optimistic Actor-Criticcited by 88
  11. 2020Discovering Reinforcement Learning Algorithmscited by 75
  12. 2020Model Selection in Contextual Stochastic Bandit Problemscited by 71
  13. 2019Improving Generalization in Meta Reinforcement Learning using Learned Objectivescited by 68
  14. 2018Efficient Bias-Span-Constrained Exploration-Exploitation in Reinforcement Learningcited by 65
  15. 2019Statistics and Samples in Distributional Reinforcement Learningcited by 62
  16. 2020Predictive Information Accelerates Learning in RLcited by 56
  17. 2021Evolving Reinforcement Learning Algorithmscited by 48
  18. 2020Regret Bound Balancing and Elimination for Model Selection in Bandits and RLcited by 45
  19. 2019Learning to Score Behaviors for Guided Policy Optimizationcited by 34
  20. 2020Ready Policy One: World Building Through Active Learningcited by 33
  21. 2020Efficient Wasserstein Natural Gradients for Reinforcement Learningcited by 23
  22. 2019Near-optimal Optimistic Reinforcement Learning using Empirical Bernstein Inequalitiescited by 15
  23. 2018QUOTA: The Quantile Option Architecture for Reinforcement Learningcited by 13
  24. 2019Kernelized Wasserstein Natural Gradientcited by 12
  25. 2019Adapting Behaviour for Learning Progresscited by 11
  26. 2019Adaptive Temporal-Difference Learning for Policy Evaluation with Per-State Uncertainty Estimatescited by 6
  27. 2021OffCon3: What is state of the art anyway?cited by 5
  28. 2019A Scheme for Dynamic Risk-Sensitive Sequential Decision Makingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 7; 2023: 13; 2024: 17; 2025: 7; 2026: 6.

55 papers.

Sort:

By area:

  1. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  2. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 104
  3. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learningcited by 50
  4. 2022Stabilizing Off-Policy Deep Reinforcement Learning from Pixelscited by 32
  5. 2022Towards Deployment-Efficient Reinforcement Learning: Lower Bound and Optimalitycited by 24
  6. 2025Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learnerscited by 23
  7. 2023Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Criticcited by 20
  8. 2021Learning Pessimism for Robust and Efficient Off-Policy Reinforcement Learningcited by 18
  9. 2024Dissecting Deep RL with High Update Ratios: Combatting Value Divergencecited by 17
  10. 2024MAD-TD: Model-Augmented Data stabilizes High Update Ratio RLcited by 16
  11. 2021A First-Occupancy Representation for Reinforcement Learningcited by 14
  12. 2024Stabilizing Reinforcement Learning in Differentiable Multiphysics Simulationcited by 14
38 more
  1. 2022Exploring through Random Curiosity with General Value Functionscited by 10
  2. 2024Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RLcited by 10
  3. 2022Robust Reinforcement Learning with Distributional Risk-averse formulationcited by 9
  4. 2025A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Controlcited by 9
  5. 2022DNS: Determinantal Point Process Based Neural Network Sampler for Ensemble Reinforcement Learningcited by 8
  6. 2021Value Activation for Bias Alleviation: Generalized-activated Deep Double Deterministic Policy Gradientscited by 6
  7. 2023On the Theory of Risk-Aware Agents: Bridging Actor-Critic and Economicscited by 6
  8. 2024Beyond Optimism: Exploration With Partially Observable Rewardscited by 6
  9. 2025The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learningcited by 6
  10. 2025Relative Entropy Pathwise Policy Optimizationcited by 6
  11. 2022Design of experiments for the calibration of history-dependent models via deep reinforcement learning and an enhanced Kalman filtercited by 5
  12. 2023Wasserstein Actor-Critic: Directed Exploration via Optimism for Continuous-Actions Controlcited by 5
  13. 2023Value-Distributional Model-Based Reinforcement Learningcited by 5
  14. 2024A Natural Extension To Online Algorithms For Hybrid RL With Limited Coveragecited by 5
  15. 2025IL-SOAR : Imitation Learning with Soft Optimistic Actor cRiticcited by 5
  16. 2023Local Optimization Achieves Global Optimality in Multi-Agent Reinforcement Learningcited by 4
  17. 2024Surprise-Adaptive Intrinsic Motivation for Unsupervised Reinforcement Learningcited by 4
  18. 2023PAC-Bayesian Soft Actor-Critic Learningcited by 3
  19. 2023A State Representation for Diminishing Rewardscited by 3
  20. 2023Pitfall of Optimism: Distributional Reinforcement Learning by Randomizing Risk Criterioncited by 3
  21. 2023Model-Based Epistemic Variance of Values for Risk-Aware Policy Optimizationcited by 3
  22. 2023Efficient Reinforcement Learning via Decoupling Exploration and Utilizationcited by 3
  23. 2024Learning Uncertainty-Aware Temporally-Extended Actionscited by 3
  24. 2024Offline Reinforcement Learning with Behavioral Supervisor Tuningcited by 3
  25. 2025Optimistic Exploration for Risk-Averse Constrained Reinforcement Learningcited by 3
  26. 2021Automating Control of Overestimation Bias for Reinforcement Learningcited by 2
  27. 2022An Experimental Comparison Between Temporal Difference and Residual Gradient with Neural Network Approximationcited by 2
  28. 2024RRLS : Robust Reinforcement Learning Suitecited by 2
  29. 2026Temporal Difference Learning with Constrained Initial Representationscited by 2
  30. 2021Multi-Agent Reinforcement Learning via Adaptive Kalman Temporal Difference and Successor Representationcited by 1
  31. 2024Bootstrapping Expectiles in Reinforcement Learningcited by 1
  32. 2025Wasserstein Barycenter Soft Actor-Criticcited by 1
  33. 2026Debiased Model-based Representations for Sample-efficient Continuous Controlcited by 1
  34. 2026Behavior-Consistent Deep Reinforcement Learningcited by 1
  35. 2023An Autonomous Non-monolithic Agent with Multi-mode Exploration based on Options Frameworkcited by 0
  36. 2023DRL-ORA: Distributional Reinforcement Learning with Online Risk Adaptioncited by 0
  37. 2024A Case for Validation Buffer in Pessimistic Actor-Criticcited by 0
  38. 2024Improving Actor-Critic Training with Steerable Action-Value Approximation Errorscited by 0

50 most cited of 55 All 55 (JSON)