mlsanityData to 7 Oct 2026

VIPO: Value Function Inconsistency Penalized Offline Reinforcement Learning

arXiv 2504.11944 ↗First posted 16 Apr 2025cs.LG cs.AIcited by 2

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) learns effective policies from pre-collected datasets, offering a practical solution for applications where online interactions are risky or costly. Model-based approaches are particularly advantageous for offline RL, owing to their data efficiency and generalizability. However, due to inherent model errors, model-based methods often artificially introduce conservatism guided by heuristic uncertainty estimation, which can be unreliable. In this paper, we introduce VIPO, a novel model-based offline RL algorithm that incorporates self-supervised feedback from value estimation to enhance model training. Specifically, the model is learned by additionally minimizing the inconsistency between the value learned directly from the offline data and the value estimated from the model. We perform comprehensive evaluations from multiple perspectives to show that VIPO can learn a highly accurate model efficiently and consistently outperform existing methods. In particular, it achieves state-of-the-art performance on almost all tasks in both D4RL and NeoRL benchmarks. Overall, VIPO offers a general framework that can be readily integrated into existing model-based offline RL algorithms to systematically enhance model accuracy.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 32 of its 48 references that are in the map. First posted on arXiv 16 Apr 2025.

Builds on

32 of 48 references.

Closest ancestors · 17

Same direction, newest first

  1. 2024Model-based Offline Reinforcement Learning with Lower Expectile Q-Learning30 Jun
  2. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning26 Apr
  3. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  4. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  5. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  6. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning8 Oct
  7. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  8. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  9. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  10. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning1 Feb
  11. 2020Offline Reinforcement Learning from Images with Latent Space Models21 Dec
  12. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
5 more
  1. 2020MOPO: Model-based Offline Policy Optimization27 May
  2. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  5. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 15

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  3. 2016OpenAI Gymcited by 1,684
  4. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  5. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  8. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  9. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  10. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  11. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  12. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
3 more
  1. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
  2. 2020Trajectory-wise Multiple Choice Learning for Dynamics Generalization in Reinforcement Learningcited by 34
  3. 2023IQL-TD-MPC: Implicit Q-Learning for Hierarchical Model Predictive Controlcited by 14

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Taming OOD Actions for Offline Reinforcement Learning: An Advantage-Based Approachcited by 1
  2. 2025Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatismcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2020POPO: Pessimistic Offline Policy Optimization
  2. 2023A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning
  3. 2026Model-based Offline RL via Robust Value-Aware Model Learning with Implicitly Differentiable Adaptive Weighting
  4. 2022Confidence-Conditioned Value Functions for Offline Reinforcement Learning
  5. 2026Robo-ValueRL: Reliable Value Estimation for Offline-to-Online Reinforcement Learning
  6. 2021Offline Reinforcement Learning with Value-based Episodic Memory
  7. 2025Use the Online Network If You Can: Towards Fast and Stable Reinforcement Learning
  8. 2024Diverse Randomized Value Functions: A Provably Pessimistic Approach for Offline Reinforcement Learning