mlsanityData to 14 Oct 2026

Offline RL With Realistic Datasets: Heteroskedasticity and Support Constraints

arXiv 2211.01052 ↗First posted 2 Nov 2022cs.LG stat.MLcited by 13

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) learns policies entirely from static datasets, thereby avoiding the challenges associated with online data collection. Practical applications of offline RL will inevitably require learning from datasets where the variability of demonstrated behaviors changes non-uniformly across the state space. For example, at a red light, nearly all human drivers behave similarly by stopping, but when merging onto a highway, some drivers merge quickly, efficiently, and safely, while many hesitate or merge dangerously. Both theoretically and empirically, we show that typical offline RL methods, which are based on distribution constraints fail to learn from data with such non-uniform variability, due to the requirement to stay close to the behavior policy to the same extent across the state space. Ideally, the learned policy should be free to choose per state how closely to follow the behavior policy to maximize long-term return, as long as the learned policy stays within the support of the behavior policy. To instantiate this principle, we reweight the data distribution in conservative Q-learning (CQL) to obtain an approximate support constraint formulation. The reweighted distribution is a mixture of the current policy and an additional policy trained to mine poor actions that are likely under the behavior policy. Our method, CQL (ReDS), is simple, theoretically motivated, and improves performance across a wide range of offline RL problems in Atari games, navigation, and pixel-based manipulation.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 46 of its 54 references that are in the map. First posted on arXiv 2 Nov 2022.

Builds on

46 of 54 references.

Closest ancestors · 21

Same direction, newest first

  1. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning31 Jan
  2. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  3. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  4. 2021Offline RL Without Off-Policy Evaluation16 Jun
  5. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  6. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  7. 2021Offline Reinforcement Learning as Anti-Exploration11 Jun
  8. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  9. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  10. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  11. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  12. 2020Offline Reinforcement Learning from Images with Latent Space Models21 Dec
9 more
  1. 2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learning12 Aug
  2. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  3. 2020Critic Regularized Regression26 Jun
  4. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  5. 2020MOPO: Model-based Offline Policy Optimization27 May
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  8. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  9. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 25

  1. 2015Continuous control with deep reinforcement learningcited by 3,407
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  7. 2017Constrained Policy Optimizationcited by 983
  8. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  9. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  10. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  11. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 725
  12. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
13 more
  1. 2021Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Datasetcited by 537
  2. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  3. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  4. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 250
  5. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 188
  6. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 144
  7. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  8. 2020Model-Based Offline Planningcited by 129
  9. 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 122
  10. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  11. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
  12. 2019Diagnosing Bottlenecks in Deep Q-learning Algorithmscited by 73
  13. 2020Heteroskedastic and Imbalanced Deep Learning with Adaptive Regularizationcited by 49

Built on by

Cited by year

Citing papers per year: 2023: 7; 2024: 5; 2025: 1.

13 papers.

Sort:

By area:

  1. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 131
  2. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
  3. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasetscited by 24
  4. 2024A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspectivecited by 11
  5. 2023Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Biascited by 7
  6. 2023Decoupled Prioritized Resampling for Offline RLcited by 6
  7. 2023Offline Reinforcement Learning with Imbalanced Datasetscited by 6
  8. 2025Dichotomous Diffusion Policy Optimizationcited by 6
  9. 2024Scaling Vision-and-Language Navigation With Offline RLcited by 5
  10. 2024Offline Reinforcement Learning with Behavioral Supervisor Tuningcited by 3
  11. 2024Decision-Point Guided Safe Policy Improvementcited by 2
  12. 2023Mildly Constrained Evaluation Policy for Offline Reinforcement Learningcited by 1
1 more
  1. 2024Augmenting Offline RL with Unlabeled Datacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Instabilities of Offline RL with Pre-Trained Neural Representation
  2. 2022Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning
  3. 2020Offline Reinforcement Learning Hands-On
  4. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting
  5. 2022Leveraging Offline Data in Online Reinforcement Learning
  6. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian
  7. 2024Out-of-Distribution Adaptation in Offline RL: Counterfactual Reasoning via Causal Normalizing Flows
  8. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learning