mlsanityData to 14 Oct 2026

CARE-VI: Conservative Adaptive Reliability Estimation for Value Improvement in Off-Policy Actor-Critic Learning

arXiv 2609.20098 ↗First posted 17 Sep 2026cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Reliable temporal-difference targets are central to off-policy actor-critic learning. Direct value improvement refines the next-state target with alternative actions, but the reliability of this refinement depends on how candidate actions are ranked, reviewed, and weighted. Noisy rankings may force premature candidate commitment, reusing selection scores may bias target valuation, and fixed enhancement weights may amplify weak evidence. To address these risks, we develop Conservative Adaptive Ranking and Screening (CARS), which retains an ordered candidate prefix within a preset budget and narrows it only when the observed boundary gap exceeds a disagreement-scaled uncertainty radius. Selector-Evaluator Value Assessment (SEVA) uses selector critics to order candidates and a separately parameterized evaluator critic to review the selected value, then caps the reviewed value at the selector reference. Dynamic Adaptive Risk-aware Enhancement (DARE) then regulates each residual correction using candidate reliability, the gap between selector and evaluator signals, and a finite stage factor. Together, CARS, SEVA, and DARE form CARE-VI, an evidence-regulated target construction framework that preserves the backbone interfaces for critic regression and actor updates. The analysis bounds the CARS boundary error, the SEVA selected-value overestimation, and the one-sided deviation of the DARE residual displacement from its population counterpart, and establishes fixed-policy recovery after the finite-stage perturbation ends. Experiments with SAC, TD3, and TD7 on four MuJoCo tasks show that CARE-VI achieves the highest mean return in all twelve settings. Grouped ablations and scalar diagnostics support the roles of the three components in improving target reliability.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 33 of its 47 references that are in the map. First posted on arXiv 17 Sep 2026.

Builds on

33 of 47 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Value Improved Actor Critic Algorithms3 Jun
  2. 2024CTD4 -- A Deep Continuous Distributional Actor-Critic Agent with a Kalman Fusion of Multiple Critics4 May

Other references · 31

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  6. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  7. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  10. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  11. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  12. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
19 more
  1. 2016Deep Exploration via Bootstrapped DQNcited by 527
  2. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  3. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 371
  4. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  5. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 302
  6. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  7. 2019A Theory of Regularized Markov Decision Processescited by 286
  8. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 219
  9. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 168
  10. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 165
  11. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 152
  12. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 126
  13. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
  14. 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
  15. 2019Fully Parameterized Quantile Function for Distributional Reinforcement Learningcited by 88
  16. 2017The Uncertainty Bellman Equation and Explorationcited by 82
  17. 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 75
  18. 2018An Analysis of Categorical Distributional Reinforcement Learningcited by 68
  19. 2023Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Criticcited by 20

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Functional Critics Are Essential for Actor-Critic: From Off-Policy Stability to Efficient Exploration
  2. 2026Stochastic Actor-Critic: Mitigating Overestimation via Temporal Aleatoric Uncertainty
  3. 2026Actor-Critic Algorithm for Dynamic Expectile and CVaR
  4. 2026Privy to the Foil: Recasting Value Estimation with a Self-Privileged Critic for RLVR
  5. 2026Collaborative Weighting with Pessimistic Critic for Mitigating Overestimation in Off-Policy Reinforcement Learning
  6. 2026Policy as Data: Replay-Based Policy Dual Averaging via Advantage Regression
  7. 2024Improving Actor-Critic Training with Steerable Action-Value Approximation Errors
  8. 2025Adaptive Ensemble Aggregation for Actor-Critics