mlsanityData to 14 Oct 2026

HIPODE: Enhancing Offline Reinforcement Learning with High-Quality Synthetic Data from a Policy-Decoupled Approach

arXiv 2306.06329 ↗First posted 10 Jun 2023cs.LGcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (ORL) has gained attention as a means of training reinforcement learning models using pre-collected static data. To address the issue of limited data and improve downstream ORL performance, recent work has attempted to expand the dataset's coverage through data augmentation. However, most of these methods are tied to a specific policy (policy-dependent), where the generated data can only guarantee to support the current downstream ORL policy, limiting its usage scope on other downstream policies. Moreover, the quality of synthetic data is often not well-controlled, which limits the potential for further improving the downstream policy. To tackle these issues, we propose HIgh-quality POlicy-DEcoupled~(HIPODE), a novel data augmentation method for ORL. On the one hand, HIPODE generates high-quality synthetic data by selecting states near the dataset distribution with potentially high value among candidate states using the negative sampling technique. On the other hand, HIPODE is policy-decoupled, thus can be used as a common plug-in method for any downstream ORL process. We conduct experiments on the widely studied TD3BC and CQL algorithms, and the results show that HIPODE outperforms the state-of-the-art policy-decoupled data augmentation method and most prevalent model-based ORL methods on D4RL benchmarks.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 32 of its 42 references that are in the map. First posted on arXiv 10 Jun 2023.

Builds on

32 of 42 references.

Closest ancestors · 21

Same direction, newest first

  1. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
  2. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief13 Oct
  3. 2022Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imagination16 Jun
  4. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning26 Apr
  5. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  6. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization9 Dec
  7. 2021d3rlpy: An Offline Deep Reinforcement Learning Library6 Nov
  8. 2021Koopman Q-learning: Offline Reinforcement Learning via Symmetries of Dynamics2 Nov
  9. 2021Offline Reinforcement Learning with Value-based Episodic Memory19 Oct
  10. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  11. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning8 Oct
  12. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
9 more
  1. 2021Offline Reinforcement Learning with Reverse Model-based Imagination1 Oct
  2. 2021Conservative Offline Distributional Reinforcement Learning12 Jul
  3. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  4. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  5. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning10 Mar
  6. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  7. 2020MOPO: Model-based Offline Policy Optimization27 May
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  9. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 11

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  6. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  7. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  8. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  9. 2020Reinforcement Learning in Economics and Financecited by 58
  10. 2022Bootstrapped Transformer for Offline Reinforcement Learningcited by 37
  11. 2019Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Samplingcited by 8

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategiescited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Model-Based Offline Reinforcement Learning with Adversarial Data Augmentation
  2. 2023Equivariant Data Augmentation for Generalization in Offline Reinforcement Learning
  3. 2024Augmenting Offline Reinforcement Learning with State-only Interactions
  4. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning
  5. 2023Small Dataset, Big Gains: Enhancing Reinforcement Learning by Offline Pre-Training with Model Based Augmentation
  6. 2024GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learning
  7. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  8. 2022Latent-Variable Advantage-Weighted Policy Optimization for Offline RL