mlsanityData to 14 Oct 2026

Offline Policy Optimization in RL with Variance Regularizaton

arXiv 2212.14405 ↗First posted 29 Dec 2022cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Learning policies from fixed offline datasets is a key challenge to scale up reinforcement learning (RL) algorithms towards practical applications. This is often because off-policy RL algorithms suffer from distributional shift, due to mismatch between dataset and the target policy, leading to high variance and over-estimation of value functions. In this work, we propose variance regularization for offline RL algorithms, using stationary distribution corrections. We show that by using Fenchel duality, we can avoid double sampling issues for computing the gradient of the variance regularizer. The proposed algorithm for offline variance regularization (OVAR) can be used to augment any existing offline policy optimization algorithms. We show that the regularizer leads to a lower bound to the offline policy optimization objective, which can help avoid over-estimation errors, and explains the benefits of our approach across a range of continuous control domains when compared to existing state-of-the-art algorithms.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 33 of its 59 references that are in the map. First posted on arXiv 29 Dec 2022.

Builds on

33 of 59 references.

Closest ancestors · 3

Same direction, newest first

  1. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  2. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  3. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 30

  1. 2022Generative Adversarial Networkscited by 5,627
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2021Convex optimizationcited by 2,515
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  7. 2016Generative Adversarial Imitation Learningcited by 1,597
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  9. 2017Constrained Policy Optimizationcited by 983
  10. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  11. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  12. 2017Deep Reinforcement Learning that Matterscited by 754
18 more
  1. 2018Reward Constrained Policy Optimizationcited by 417
  2. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  3. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  4. 2019Batch Policy Learning under Constraintscited by 244
  5. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 227
  6. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 188
  7. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 183
  8. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  9. 2020Provably Efficient Safe Exploration via Primal-Dual Policy Optimizationcited by 134
  10. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  11. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  12. 2014Algorithms for CVaR Optimization in MDPscited by 99
  13. 2019Risk-Averse Trust Region Optimization for Reward-Volatility Reductioncited by 54
  14. 2018Policy Optimization via Importance Samplingcited by 47
  15. 2020Experience Replay with Likelihood-free Importance Weightscited by 42
  16. 2014Variance-Constrained Actor-Critic Algorithms for Discounted and Average Reward MDPscited by 28
  17. 2019Generalized Off-Policy Actor-Criticcited by 16
  18. 2020Stable Policy Optimization via Off-Policy Divergence Regularizationcited by 16

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning
  2. 2024Policy-regularized Offline Multi-objective Reinforcement Learning
  3. 2021Near-Optimal Offline Reinforcement Learning via Double Variance Reduction
  4. 2023Offline Primal-Dual Reinforcement Learning for Linear MDPs
  5. 2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learning
  6. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learning
  7. 2023Offline Reinforcement Learning with On-Policy Q-Function Regularization
  8. 2026Conservative Query and Adaptive Regularization for Offline RL Under Uncertainty Estimation