mlsanityData to 7 Oct 2026

Benchmarks for Reinforcement Learning with Biased Offline Data and Imperfect Simulators

arXiv 2407.00806 ↗First posted 30 Jun 2024cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

In many reinforcement learning (RL) applications one cannot easily let the agent act in the world; this is true for autonomous vehicles, healthcare applications, and even some recommender systems, to name a few examples. Offline RL provides a way to train agents without real-world exploration, but is often faced with biases due to data distribution shifts, limited coverage, and incomplete representation of the environment. To address these issues, practical applications have tried to combine simulators with grounded offline data, using so-called hybrid methods. However, constructing a reliable simulator is in itself often challenging due to intricate system complexities as well as missing or incomplete information. In this work, we outline four principal challenges for combining offline data with imperfect simulators in RL: simulator modeling error, partial observability, state and action discrepancies, and hidden confounding. To help drive the RL community to pursue these problems, we construct ``Benchmarks for Mechanistic Offline Reinforcement Learning'' (B4MRL), which provide dataset-simulator benchmarks for the aforementioned challenges. Our results suggest the key necessity of such benchmarks for future research.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 27 of its 41 references that are in the map. First posted on arXiv 30 Jun 2024.

Builds on

27 of 41 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confounding1 Jun
  2. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learning27 Jun
  3. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  4. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  5. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 20

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  5. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  6. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  7. 2019Causal Confusion in Imitation Learningcited by 286
  8. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  9. 2023Nonparametric Identifiability of Causal Representations from Unknown Interventionscited by 105
  10. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 92
  11. 2019Off-Policy Evaluation in Partially Observable Environmentscited by 65
  12. 2018Removing Hidden Confounding by Experimental Groundingcited by 64
8 more
  1. 2019Meta Reinforcement Learning for Sim-to-real Domain Adaptationcited by 49
  2. 2020Sim2Real Transfer for Reinforcement Learning without Dynamics Randomizationcited by 43
  3. 2020Provably Efficient Causal Reinforcement Learning with Confounded Observational Datacited by 40
  4. 2020Bandits with Partially Observable Confounded Datacited by 22
  5. 2018Causal Identification under Markov Equivalencecited by 15
  6. 2021On Covariate Shift of Latent Confounders in Imitation and Reinforcement Learningcited by 14
  7. 2023A Policy Gradient Method for Confounded POMDPscited by 6
  8. 2018The Bottleneck Simulator: A Model-based Deep Reinforcement Learning Approachcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2020Offline Reinforcement Learning Hands-On
  2. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
  3. 2023Improving and Benchmarking Offline Reinforcement Learning Algorithms
  4. 2025NeoRL-2: Near Real-World Benchmarks for Offline Reinforcement Learning with Extended Realistic Scenarios
  5. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning
  6. 2024D5RL: Diverse Datasets for Data-Driven Deep Reinforcement Learning
  7. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learning
  8. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies