mlsanityData to 7 Oct 2026

Improving Offline Reinforcement Learning with Inaccurate Simulators

arXiv 2405.04307 ↗First posted 7 May 2024cs.RO cs.AI cs.LGcited by 4

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) provides a promising approach to avoid costly online interaction with the real environment. However, the performance of offline RL highly depends on the quality of the datasets, which may cause extrapolation error in the learning process. In many robotic applications, an inaccurate simulator is often available. However, the data directly collected from the inaccurate simulator cannot be directly used in offline RL due to the well-known exploration-exploitation dilemma and the dynamic gap between inaccurate simulation and the real environment. To address these issues, we propose a novel approach to combine the offline dataset and the inaccurate simulation data in a better manner. Specifically, we pre-train a generative adversarial network (GAN) model to fit the state distribution of the offline dataset. Given this, we collect data from the inaccurate simulator starting from the distribution provided by the generator and reweight the simulated data using the discriminator. Our experimental results in the D4RL benchmark and a real-world manipulation task confirm that our method can benefit more from both inaccurate simulator and limited offline datasets to achieve better performance than the state-of-the-art methods.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 35 of its 42 references that are in the map. First posted on arXiv 7 May 2024.

Builds on

35 of 42 references.

Closest ancestors · 16

Same direction, newest first

  1. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization28 Mar
  2. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learning27 Jun
  3. 2022Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imagination16 Jun
  4. 2021Importance of Empirical Sample Complexity Analysis for Offline Reinforcement Learning31 Dec
  5. 2021RvS: What is Essential for Offline RL via Supervised Learning?20 Dec
  6. 2021A Dataset Perspective on Offline Reinforcement Learning8 Nov
  7. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  8. 2021Offline Reinforcement Learning with Reverse Model-based Imagination1 Oct
  9. 2021Offline RL Without Off-Policy Evaluation16 Jun
  10. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  11. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning10 Mar
  12. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
4 more
  1. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  4. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 19

  1. 2022Generative Adversarial Networkscited by 5,603
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  6. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  8. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  9. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  10. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 388
  11. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  12. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
7 more
  1. 2018Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomizationcited by 292
  2. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  3. 2018Sim-to-Real Reinforcement Learning for Deformable Object Manipulationcited by 124
  4. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RLcited by 89
  5. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 58
  6. 2020Sim2Real Transfer for Reinforcement Learning without Dynamics Randomizationcited by 43
  7. 2022A State-Distribution Matching Approach to Non-Episodic Reinforcement Learningcited by 21

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 2.

4 papers.

  1. 2024xTED: Cross-Domain Adaptation via Diffusion-Based Trajectory Editingcited by 9
  2. 2024Provable Domain Adaptation for Offline Reinforcement Learning with Limited Samplescited by 5
  3. 2025Revealing the Challenges of Sim-to-Real Transfer in Model-Based Reinforcement Learning via Latent Space Modelingcited by 1
  4. 2025Multi-Fidelity Hybrid Reinforcement Learning via Information Gain Maximizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Benchmarks for Reinforcement Learning with Biased Offline Data and Imperfect Simulators
  2. 2025Model-Based Offline Reinforcement Learning with Adversarial Data Augmentation
  3. 2023Small Dataset, Big Gains: Enhancing Reinforcement Learning by Offline Pre-Training with Model Based Augmentation
  4. 2023Model-based Offline Policy Optimization with Adversarial Network
  5. 2023Finetuning Offline World Models in the Real World
  6. 2024Towards Robust Policy: Enhancing Offline Reinforcement Learning with Adversarial Attacks and Defenses
  7. 2023Adversarial Model for Offline Reinforcement Learning
  8. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies