mlsanityData to 7 Oct 2026

Uncertainty-driven Trajectory Truncation for Data Augmentation in Offline Reinforcement Learning

arXiv 2304.04660 ↗First posted 10 Apr 2023cs.LG cs.AIcited by 13

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Equipped with the trained environmental dynamics, model-based offline reinforcement learning (RL) algorithms can often successfully learn good policies from fixed-sized datasets, even some datasets with poor quality. Unfortunately, however, it can not be guaranteed that the generated samples from the trained dynamics model are reliable (e.g., some synthetic samples may lie outside of the support region of the static dataset). To address this issue, we propose Trajectory Truncation with Uncertainty (TATU), which adaptively truncates the synthetic trajectory if the accumulated uncertainty along the trajectory is too large. We theoretically show the performance bound of TATU to justify its benefits. To empirically show the advantages of TATU, we first combine it with two classical model-based offline RL algorithms, MOPO and COMBO. Furthermore, we integrate TATU with several off-the-shelf model-free offline RL algorithms, e.g., BCQ. Experimental results on the D4RL benchmark show that TATU significantly improves their performance, often by a large margin. Code is available here.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 38 of its 47 references that are in the map. First posted on arXiv 10 Apr 2023.

Builds on

38 of 47 references.

Closest ancestors · 25

Same direction, newest first

  1. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief13 Oct
  2. 2022A Unified Framework for Alternating Offline Model Training and Policy Learning12 Oct
  3. 2022Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imagination16 Jun
  4. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  5. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning26 Apr
  6. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  7. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  8. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  9. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  10. 2021Offline Reinforcement Learning with Reverse Model-based Imagination1 Oct
  11. 2021Conservative Offline Distributional Reinforcement Learning12 Jul
  12. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL16 Jun
13 more
  1. 2021Offline RL Without Off-Policy Evaluation16 Jun
  2. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  3. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  4. 2021Model-Based Offline Planning with Trajectory Pruning16 May
  5. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  6. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  7. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  8. 2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learning12 Aug
  9. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  10. 2020MOPO: Model-based Offline Policy Optimization27 May
  11. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  12. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  13. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 13

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2014Conditional Generative Adversarial Netscited by 2,152
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  7. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  8. 2017Deep Reinforcement Learning framework for Autonomous Drivingcited by 224
  9. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  10. 2020Model-Based Offline Planningcited by 126
  11. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  12. 2020Trust the Model When It Is Confident: Masked Model-based Actor-Criticcited by 36
1 more
  1. 2021Challenges for Reinforcement Learning in Healthcarecited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 7; 2026: 2.

13 papers.

Sort:

By area:

  1. 2024DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory Stitchingcited by 35
  2. 2024ODRL: A Benchmark for Off-Dynamics Reinforcement Learningcited by 25
  3. 2024Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learningcited by 18
  4. 2024Goal-Conditioned Data Augmentation for Offline Reinforcement Learningcited by 4
  5. 2025BiTrajDiff: Bidirectional Trajectory Generation with Diffusion Models for Offline Reinforcement Learningcited by 4
  6. 2025Hybrid Cross-domain Robust Reinforcement Learningcited by 3
  7. 2025Offline Reinforcement Learning via Inverse Optimizationcited by 2
  8. 2025Adaptive Scaling of Policy Constraints for Offline Reinforcement Learningcited by 1
  9. 2025Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatismcited by 1
  10. 2026Conservative Continuous-Time Treatment Optimizationcited by 1
  11. 2025Model-Based Offline Reinforcement Learning with Reliability-Guaranteed Sequence Modelingcited by 0
  12. 2025RAD: Retrieval High-quality Demonstrations to Enhance Decision-makingcited by 0
1 more
  1. 2026Improving Generalization and Robustness in Offline Reinforcement Learning via Boundary-Aware Data Augmentationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Offline Trajectory Optimization for Offline Reinforcement Learning
  2. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning
  3. 2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization
  4. 2022Robust Offline Reinforcement Learning with Gradient Penalty and Constraint Relaxation
  5. 2025Evaluation-Time Policy Switching for Offline Reinforcement Learning
  6. 2024SUMO: Search-Based Uncertainty Estimation for Model-Based Offline Reinforcement Learning
  7. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting
  8. 2023DOMAIN: MilDly COnservative Model-BAsed OfflINe Reinforcement Learning