mlsanityData to 7 Oct 2026

Mitigating Distribution Shift in Model-based Offline RL via Shifts-aware Reward Learning

arXiv 2408.12830 ↗First posted 23 Aug 2024cs.LG stat.MLcited by 2

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Model-based offline reinforcement learning trains policies using pre-collected datasets and learned environment models, eliminating the need for direct real-world environment interaction. However, this paradigm is inherently challenged by distribution shift~(DS). Existing methods address this issue by leveraging off-policy mechanisms and estimating model uncertainty, but they often result in inconsistent objectives and lack a unified theoretical foundation. This paper offers a comprehensive analysis that disentangles the problem into two fundamental components: model bias and policy shift. Our theoretical and empirical investigations reveal how these factors distort value estimation and restrict policy optimization. To tackle these challenges, we derive a novel shifts-aware reward through a unified probabilistic inference framework, which modifies the vanilla reward to refine value learning and facilitate policy training. Building on this, we develop a practical implementation that leverages classifier-based techniques to approximate the adjusted reward for effective policy optimization. Empirical results across multiple benchmarks demonstrate that the proposed approach mitigates distribution shift and achieves superior or comparable performance, validating our theoretical insights.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 19 of its 33 references that are in the map. First posted on arXiv 23 Aug 2024.

Builds on

19 of 33 references.

Closest ancestors · 8

Same direction, newest first

  1. 2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization7 Oct
  2. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning8 Oct
  3. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  4. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  5. 2020MOPO: Model-based Offline Policy Optimization27 May
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  8. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 11

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  5. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  6. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  7. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 152
  8. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 138
  9. 2020Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifierscited by 93
  10. 2019State Alignment-based Imitation Learningcited by 73
  11. 2023State Regularized Policy Optimization on Data with Dynamics Shiftcited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Generative OOD-regularized Model-based Policy Optimizationcited by 0
  2. 2026A Unified Causal-Origin Taxonomy of Distributional Shifts in Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Diffusion Policies for Risk-Averse Behavior Modeling in Offline Reinforcement Learning
  2. 2024Out-of-Distribution Adaptation in Offline RL: Counterfactual Reasoning via Causal Normalizing Flows
  3. 2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learning
  4. 2023A Survey on Offline Model-Based Reinforcement Learning
  5. 2023Selective Uncertainty Propagation in Offline RL
  6. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation
  7. 2023Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coverage
  8. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief