mlsanityData to 7 Oct 2026

Model-based Offline Reinforcement Learning with Local Misspecification

arXiv 2301.11426 ↗First posted 26 Jan 2023cs.LGcited by 5

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We present a model-based offline reinforcement learning policy performance lower bound that explicitly captures dynamics model misspecification and distribution mismatch and we propose an empirical algorithm for optimal offline policy selection. Theoretically, we prove a novel safe policy improvement theorem by establishing pessimism approximations to the value function. Our key insight is to jointly consider selecting over dynamics models and policies: as long as a dynamics model can accurately represent the dynamics of the state-action pairs visited by a given policy, it is possible to approximate the value of that particular policy. We analyze our lower bound in the LQR setting and also show competitive performance to previous lower bounds on policy selection across a set of D4RL tasks.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 53 of its 71 references that are in the map. First posted on arXiv 26 Jan 2023.

Builds on

53 of 71 references.

Closest ancestors · 18

Same direction, newest first

  1. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning22 Sep
  2. 2021Model-Based Offline Planning with Trajectory Pruning16 May
  3. 2021Nearly Horizon-Free Offline Reinforcement Learning25 Mar
  4. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  5. 2021Infinite-Horizon Offline Reinforcement Learning with Linear Function Approximation: Curse of Dimensionality and Algorithm17 Mar
  6. 2021Instabilities of Offline RL with Pre-Trained Neural Representation8 Mar
  7. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  8. 2021Near-Optimal Offline Reinforcement Learning via Double Variance Reduction2 Feb
  9. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  10. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?22 Oct
  11. 2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learning12 Aug
  12. 2020Batch Value-function Approximation with Only Realizability11 Aug
6 more
  1. 2020Hyperparameter Selection for Offline Reinforcement Learning17 Jul
  2. 2020Near-Optimal Provable Uniform Convergence in Offline Policy Evaluation for Reinforcement Learning7 Jul
  3. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  4. 2020MOPO: Model-based Offline Policy Optimization27 May
  5. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 35

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  3. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  4. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,315
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  7. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  8. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  9. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  10. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  11. 2019Batch Policy Learning under Constraintscited by 242
  12. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
23 more
  1. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
  2. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  3. 2020Model-Based Offline Planningcited by 126
  4. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  5. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  6. 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 118
  7. 2019When to use parametric models in reinforcement learning?cited by 118
  8. 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 118
  9. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  10. 2020Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planningcited by 83
  11. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
  12. 2020GradientDICE: Rethinking Generalized Offline Estimation of Stationary Valuescited by 70
  13. 2020Asymptotically Efficient Off-Policy Evaluation for Tabular Reinforcement Learningcited by 60
  14. 2020Exponential Lower Bounds for Batch Reinforcement Learning: Batch RL can be Exponentially Harder than Online RLcited by 55
  15. 2021Offline Model-Based Optimization via Normalized Maximum Likelihood Estimationcited by 38
  16. 2019Understanding the Curse of Horizon in Off-Policy Evaluation via Conditional Importance Samplingcited by 36
  17. 2018Deep Reinforcement Learning for Vision-Based Robotic Grasping: A Simulated Comparative Evaluation of Off-Policy Methodscited by 32
  18. 2019Calibrated Model-Based Deep Reinforcement Learningcited by 31
  19. 2021On the Optimality of Batch Policy Optimization Algorithmscited by 28
  20. 2018Representation Balancing MDPs for Off-Policy Policy Evaluationcited by 20
  21. 2021Minimax Model Learningcited by 16
  22. 2020Minimax Value Interval for Off-Policy Evaluation and Policy Optimizationcited by 11
  23. 2021Distillation of RL Policies with Formal Guarantees via Variational Abstraction of Markov Decision Processes (Technical Report)cited by 9

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2; 2025: 2.

5 papers.

  1. 2023PASTA: Pretrained Action-State Transformer Agentscited by 5
  2. 2024OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimatorscited by 3
  3. 2025Deep SPI: Safe Policy Improvement via World Modelscited by 3
  4. 2025The Bias-Variance Tradeoff in Data-Driven Optimization: A Local Misspecification Perspectivecited by 3
  5. 2024A General Control-Theoretic Approach for Reinforcement Learning: Theory and Algorithmscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning
  2. 2021DROMO: Distributionally Robust Offline Model-based Policy Optimization
  3. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief
  4. 2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization
  5. 2024The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning
  6. 2023A Survey on Offline Model-Based Reinforcement Learning
  7. 2026Offline Policy Optimization with Posterior Sampling
  8. 2025Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol