mlsanityData to 14 Oct 2026

Hybrid Value Estimation for Off-policy Evaluation and Offline Reinforcement Learning

arXiv 2206.02000 ↗First posted 4 Jun 2022cs.LGcited by 3

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Value function estimation is an indispensable subroutine in reinforcement learning, which becomes more challenging in the offline setting. In this paper, we propose Hybrid Value Estimation (HVE) to reduce value estimation error, which trades off bias and variance by balancing between the value estimation from offline data and the learned model. Theoretical analysis discloses that HVE enjoys a better error bound than the direct methods. HVE can be leveraged in both off-policy evaluation and offline reinforcement learning settings. We, therefore, provide two concrete algorithms Off-policy HVE (OPHVE) and Model-based Offline HVE (MOHVE), respectively. Empirical evaluations on MuJoCo tasks corroborate the theoretical claim. OPHVE outperforms other off-policy evaluation methods in all three metrics measuring the estimation effectiveness, while MOHVE achieves better or comparable performance with state-of-the-art offline reinforcement learning algorithms. We hope that HVE could shed some light on further research on reinforcement learning from fixed data.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 36 of its 47 references that are in the map. First posted on arXiv 4 Jun 2022.

Builds on

36 of 47 references.

Closest ancestors · 12

Same direction, newest first

  1. 2021Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters8 Oct
  2. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  3. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  4. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  5. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning1 Feb
  6. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  7. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  8. 2020MOPO: Model-based Offline Policy Optimization27 May
  9. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  10. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  11. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  12. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 24

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  5. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  6. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  7. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  8. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  9. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  10. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  11. 2020Learning Agile Robotic Locomotion Skills by Imitating Animalscited by 251
  12. 2019Batch Policy Learning under Constraintscited by 244
12 more
  1. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  2. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  3. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  4. 2020Error Bounds of Imitating Policies and Environmentscited by 90
  5. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  6. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  7. 2018Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learningcited by 80
  8. 2018Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learningcited by 71
  9. 2017Consistent On-Line Off-Policy Evaluationcited by 51
  10. 2020Statistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluationcited by 28
  11. 2021Regret Minimization Experience Replay in Off-Policy Reinforcement Learningcited by 25
  12. 2021SOPE: Spectrum of Off-Policy Estimatorscited by 5

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 1; 2024: 1.

3 papers.

  1. 2022A Survey on Model-based Reinforcement Learningcited by 66
  2. 2024Energy-Guided Diffusion Sampling for Offline-to-Online Reinforcement Learningcited by 17
  3. 2023How To Guide Your Learner: Imitation Learning with Active Adaptive Expert Involvementcited by 8

Similar, not cited

Close in wording, no citation either way

  1. 2023A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning
  2. 2023Conservative State Value Estimation for Offline Reinforcement Learning
  3. 2025Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol
  4. 2021Projected State-action Balancing Weights for Offline Reinforcement Learning
  5. 2024Diverse Randomized Value Functions: A Provably Pessimistic Approach for Offline Reinforcement Learning
  6. 2024Is Value Learning Really the Main Bottleneck in Offline RL?
  7. 2026Peng's Q(λ) for Conservative Value Estimation in Offline Reinforcement Learning
  8. 2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization