mlsanityData to 7 Oct 2026

A Case for Validation Buffer in Pessimistic Actor-Critic

arXiv 2403.01014 ↗First posted 1 Mar 2024cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In this paper, we investigate the issue of error accumulation in critic networks updated via pessimistic temporal difference objectives. We show that the critic approximation error can be approximated via a recursive fixed-point model similar to that of the Bellman value. We use such recursive definition to retrieve the conditions under which the pessimistic critic is unbiased. Building on these insights, we propose Validation Pessimism Learning (VPL) algorithm. VPL uses a small validation buffer to adjust the levels of pessimism throughout the agent training, with the pessimism set such that the approximation error of the critic targets is minimized. We investigate the proposed approach on a variety of locomotion and manipulation tasks and report improvements in sample efficiency and performance.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 47 of its 71 references that are in the map. First posted on arXiv 1 Mar 2024.

Builds on

47 of 71 references.

References · 47

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2016Layer Normalizationcited by 5,579
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  5. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  7. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  8. 2018DeepMind Control Suitecited by 935
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  10. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  11. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  12. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
35 more
  1. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  2. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  3. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  4. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  5. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  6. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  7. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 202
  8. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  9. 2023Understanding plasticity in neural networkscited by 170
  10. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
  11. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 157
  12. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 154
  13. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
  14. 2021Discovering and Achieving Goals via World Modelscited by 131
  15. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 108
  16. 2018Lipschitz Continuity in Model-based Reinforcement Learningcited by 90
  17. 2019Better Exploration with Optimistic Actor-Criticcited by 88
  18. 2016Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Unitscited by 86
  19. 2020DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correctioncited by 71
  20. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 64
  21. 2019Statistics and Samples in Distributional Reinforcement Learningcited by 62
  22. 2021Tactical Optimism and Pessimism for Deep Reinforcement Learningcited by 55
  23. 2023PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learningcited by 42
  24. 2020Trust the Model When It Is Confident: Masked Model-based Actor-Criticcited by 36
  25. 2023Efficient Deep Reinforcement Learning Requires Regulating Overfittingcited by 36
  26. 2020Ready Policy One: World Building Through Active Learningcited by 33
  27. 2023An Analysis of Quantile Temporal-Difference Learningcited by 30
  28. 2018Expected Policy Gradients for Reinforcement Learningcited by 28
  29. 2021Sample-Efficient Reinforcement Learning via Conservative Model-Based Actor-Criticcited by 23
  30. 2021Learning Pessimism for Robust and Efficient Off-Policy Reinforcement Learningcited by 18
  31. 2021Sample Efficient Reinforcement Learning via Model-Ensemble Exploration and Exploitationcited by 17
  32. 2020Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via Latent Model Ensemblescited by 14
  33. 2021Continuous Control With Ensemble Deep Deterministic Policy Gradientscited by 8
  34. 2023On the Theory of Risk-Aware Agents: Bridging Actor-Critic and Economicscited by 6
  35. 2021Automating Control of Overestimation Bias for Reinforcement Learningcited by 2

Built on by

Not cited yet.