mlsanityData to 7 Oct 2026

Optimality Inductive Biases and Agnostic Guidelines for Offline Reinforcement Learning

arXiv 2107.01407 ↗First posted 3 Jul 2021cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

The performance of state-of-the-art offline RL methods varies widely over the spectrum of dataset qualities, ranging from far-from-optimal random data to close-to-optimal expert demonstrations. We re-implement these methods to test their reproducibility, and show that when a given method outperforms the others on one end of the spectrum, it never does on the other end. This prevents us from naming a victor across the board. We attribute the asymmetry to the amount of inductive bias injected into the agent to entice it to posit that the behavior underlying the offline dataset is optimal for the task. Our investigations confirm that careless injections of such optimality inductive biases make dominant agents subpar as soon as the offline policy is sub-optimal. To bridge this gap, we generalize importance-weighted regression methods that have proved the most versatile across the spectrum of dataset grades into a modular framework that allows for the design of methods that align with how much we know about the dataset. This modularity enables qualitatively different injections of optimality inductive biases. We show that certain orchestrations strike the right balance, improving the return on one end of the spectrum without harming it on the other end. While the formulation of guidelines for the design of an offline method reduces to aligning the amount of optimality bias to inject with what we know about the quality of the data, the design of an agnostic method for which we need not know the quality of the data beforehand is more nuanced. Only our framework allowed us to design a method that performed well across the spectrum while remaining modular if more information about the quality of the data ever becomes available.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 14 of its 203 references that are in the map. First posted on arXiv 3 Jul 2021.

Builds on

14 of 203 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 13

  1. 2020VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representationcited by 528
  2. 2017End-to-end Driving via Conditional Imitation Learningcited by 479
  3. 2015DeepDriving: Learning Affordance for Direct Perception in Autonomous Drivingcited by 229
  4. 2017Deep Reinforcement Learning framework for Autonomous Drivingcited by 224
  5. 2018Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networkscited by 171
  6. 2017Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Graspingcited by 138
  7. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  8. 2020SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Drivingcited by 121
  9. 2019IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Datacited by 85
  10. 2020NeuralSim: Augmenting Differentiable Simulators with Neural Networkscited by 77
  11. 2018Deep Reinforcement Learning for Vision-Based Robotic Grasping: A Simulated Comparative Evaluation of Off-Policy Methodscited by 32
  12. 2017A Fast Integrated Planning and Control Framework for Autonomous Driving via Imitation Learningcited by 20
1 more
  1. 2018Learning End-to-end Autonomous Driving using Guided Auxiliary Supervisioncited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Generalization in offline RL: The structure is more important than the amount of pessimism
  2. 2025Expert or not? assessing data quality in offline reinforcement learning
  3. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
  4. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
  5. 2025On the Statistical Complexity for Offline and Low-Adaptive Reinforcement Learning with Structures
  6. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  7. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  8. 2023Survival Instinct in Offline Reinforcement Learning