mlsanityData to 14 Oct 2026

Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting

arXiv 2306.13085 ↗First posted 22 Jun 2023cs.LG cs.AIcited by 22

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Most offline reinforcement learning (RL) algorithms return a target policy maximizing a trade-off between (1) the expected performance gain over the behavior policy that collected the dataset, and (2) the risk stemming from the out-of-distribution-ness of the induced state-action occupancy. It follows that the performance of the target policy is strongly related to the performance of the behavior policy and, thus, the trajectory return distribution of the dataset. We show that in mixed datasets consisting of mostly low-return trajectories and minor high-return trajectories, state-of-the-art offline RL algorithms are overly restrained by low-return trajectories and fail to exploit high-performing trajectories to the fullest. To overcome this issue, we show that, in deterministic MDPs with stochastic initial states, the dataset sampling can be re-weighted to induce an artificial dataset whose behavior policy has a higher return. This re-weighted sampling strategy may be combined with any offline RL algorithm. We further analyze that the opportunity for performance improvement over the behavior policy correlates with the positive-sided variance of the returns of the trajectories in the dataset. We empirically show that while CQL, IQL, and TD3+BC achieve only a part of this potential policy improvement, these same algorithms combined with our reweighted sampling strategy fully exploit the dataset. Furthermore, we empirically demonstrate that, despite its theoretical limitation, the approach may still be efficient in stochastic environments. The code is available at https://github.com/Improbable-AI/harness-offline-rl.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 33 of its 48 references that are in the map. First posted on arXiv 22 Jun 2023.

Builds on

33 of 48 references.

Closest ancestors · 7

Same direction, newest first

  1. 2022When does return-conditioned supervised learning work for offline reinforcement learning?2 Jun
  2. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation21 Nov
  3. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  4. 2021Offline RL Without Off-Policy Evaluation16 Jun
  5. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  6. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 26

  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  2. 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,245
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  4. 2015Prioritized Experience Replaycited by 1,099
  5. 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 989
  6. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  7. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  8. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  9. 2018Behavioral Cloning from Observationcited by 492
  10. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  11. 2018Distributed Prioritized Experience Replaycited by 269
  12. 2021Delving into Deep Imbalanced Regressioncited by 158
14 more
  1. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 144
  2. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 125
  3. 2019Training Agents using Upside-Down Reinforcement Learningcited by 107
  4. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 106
  5. 2019Reward-Conditioned Policiescited by 102
  6. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  7. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
  8. 2020DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correctioncited by 71
  9. 2018Imbalanced Deep Learning by Minority Class Incremental Rectificationcited by 65
  10. 2016Safe Policy Improvement by Minimizing Robust Baseline Regretcited by 53
  11. 2022Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matchingcited by 43
  12. 2020Experience Replay with Likelihood-free Importance Weightscited by 42
  13. 2022Topological Experience Replaycited by 17
  14. 2021Large Batch Experience Replaycited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 8; 2025: 9; 2026: 2.

22 papers.

Sort:

By area:

  1. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasetscited by 24
  2. 2024GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learningcited by 22
  3. 2024Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learningcited by 13
  4. 2024A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspectivecited by 11
  5. 2023Understanding and Addressing the Pitfalls of Bisimulation-based Representations in Offline Reinforcement Learningcited by 10
  6. 2024OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learningcited by 10
  7. 2025GPO: Learning from Critical Steps to Improve LLM Reasoningcited by 10
  8. 2024Learning from Sparse Offline Datasets via Conservative Density Estimationcited by 9
  9. 2025Behavior Injection: Preparing Language Models for Reinforcement Learningcited by 8
  10. 2023Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Biascited by 7
  11. 2024Simple Ingredients for Offline Reinforcement Learningcited by 4
  12. 2025Fewer May Be Better: Enhancing Offline Reinforcement Learning with Reduced Datasetcited by 4
10 more
  1. 2025An Optimal Discriminator Weighted Imitation Perspective for Reinforcement Learningcited by 4
  2. 2025Tailored Primitive Initialization is the Secret Key to Reinforcement Learningcited by 4
  3. 2024Deterministic Uncertainty Propagation for Improved Model-Based Offline Reinforcement Learningcited by 2
  4. 2024HG2P: Hippocampus-inspired High-reward Graph and Model-Free Q-Gradient Penalty for Path Planning and Motion Controlcited by 1
  5. 2025Adaptive Scaling of Policy Constraints for Offline Reinforcement Learningcited by 1
  6. 2026SPAR: Support-Preserving Action Rectificationcited by 1
  7. 2025Beyond the Proxy: Trajectory-Distilled Guidance for Offline GFlowNet Trainingcited by 0
  8. 2025CAWR: Corruption-Averse Advantage-Weighted Regression for Robust Policy Optimizationcited by 0
  9. 2025An Empirical Study on the Effectiveness of Incorporating Offline RL As Online RL Subroutinescited by 0
  10. 2026Trajectory-Level Data Augmentation for Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Prioritized Trajectory Replay: A Replay Memory for Data-driven Reinforcement Learning
  2. 2021A Dataset Perspective on Offline Reinforcement Learning
  3. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  4. 2022Boosting Offline Reinforcement Learning via Data Rebalancing
  5. 2025Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Model
  6. 2022Semi-Supervised Offline Reinforcement Learning with Action-Free Trajectories
  7. 2021Behavior Constraining in Weight Space for Offline Reinforcement Learning
  8. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond