mlsanityData to 7 Oct 2026

Model-Based Offline Reinforcement Learning with Reliability-Guaranteed Sequence Modeling

arXiv 2502.06491 ↗First posted 10 Feb 2025cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Model-based offline reinforcement learning (MORL) aims to learn a policy by exploiting a dynamics model derived from an existing dataset. Applying conservative quantification to the dynamics model, most existing works on MORL generate trajectories that approximate the real data distribution to facilitate policy learning by using current information (e.g., the state and action at time step t). However, these works neglect the impact of historical information on environmental dynamics, leading to the generation of unreliable trajectories that may not align with the real data distribution. In this paper, we propose a new MORL algorithm Reliability-guaranteed Transformer (RT), which can eliminate unreliable trajectories by calculating the cumulative reliability of the generated trajectory (i.e., using a weighted variational distance away from the real data). Moreover, by sampling candidate actions with high rewards, RT can efficiently generate high-return trajectories from the existing offline data. We theoretically prove the performance guarantees of RT in policy learning, and empirically demonstrate its effectiveness against state-of-the-art model-based methods on several benchmark tasks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 35 of its 44 references that are in the map. First posted on arXiv 10 Feb 2025.

Builds on

35 of 44 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory Stitching4 Feb
  2. 2023Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coverage16 May
  3. 2023Uncertainty-driven Trajectory Truncation for Data Augmentation in Offline Reinforcement Learning10 Apr
  4. 2022Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imagination16 Jun
  5. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  6. 2021Offline Reinforcement Learning with Reverse Model-based Imagination1 Oct
  7. 2021Model-Based Offline Planning with Trajectory Pruning16 May
  8. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  9. 2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learning12 Aug
  10. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  11. 2020MOPO: Model-based Offline Policy Optimization27 May
  12. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
1 more
  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 22

  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  3. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  4. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  5. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  6. 2018Behavioral Cloning from Observationcited by 482
  7. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 307
  8. 2022Multi-Game Decision Transformerscited by 212
  9. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
  10. 2023Synthetic Experience Replaycited by 100
  11. 2018Lipschitz Continuity in Model-based Reinforcement Learningcited by 90
  12. 2016Self-Correcting Models for Model-Based Reinforcement Learningcited by 40
10 more
  1. 2023ACT: Empowering Decision Transformer with Dynamic Programming via Advantage Conditioningcited by 24
  2. 2021UMBRELLA: Uncertainty-Aware Model-Based Offline Reinforcement Learning Leveraging Planningcited by 23
  3. 2023Waypoint Transformer: Reinforcement Learning via Supervised Learning with Intermediate Targetscited by 22
  4. 2023Conditional Variational Autoencoder for Sign Language Translation with Cross-Modal Alignmentcited by 22
  5. 2023Is Model Ensemble Necessary? Model-based RL via a Single Model with Lipschitz Regularized Value Functioncited by 19
  6. 2023Optimal Exploration for Model-Based RL in Nonlinear Systemscited by 17
  7. 2023Model-Based Uncertainty in Value Functionscited by 15
  8. 2024Offline Imitation Learning with Model-based Reverse Augmentationcited by 8
  9. 2023Data-Efficient Task Generalization via Probabilistic Model-based Meta Reinforcement Learningcited by 6
  10. 2022Bridging the Data Gap between Training and Inference for Unsupervised Neural Machine Translationcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Model-based Offline Reinforcement Learning with Local Misspecification
  2. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning
  3. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity
  4. 2025Policy-Driven World Model Adaptation for Robust Offline Model-based Reinforcement Learning
  5. 2024Offline Trajectory Optimization for Offline Reinforcement Learning
  6. 2024The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning
  7. 2023DOMAIN: MilDly COnservative Model-BAsed OfflINe Reinforcement Learning
  8. 2023On the Sample Complexity of Vanilla Model-Based Offline Reinforcement Learning with Dependent Samples