mlsanityData to 13 Oct 2026

User-Interactive Offline Reinforcement Learning

arXiv 2205.10629 ↗First posted 21 May 2022cs.LGcited by 11

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning algorithms still lack trust in practice due to the risk that the learned policy performs worse than the original policy that generated the dataset or behaves in an unexpected way that is unfamiliar to the user. At the same time, offline RL algorithms are not able to tune their most important hyperparameter - the proximity of the learned policy to the original policy. We propose an algorithm that allows the user to tune this hyperparameter at runtime, thereby addressing both of the above mentioned issues simultaneously. This allows users to start with the original behavior and grant successively greater deviation, as well as stopping at any time when the policy deteriorates or the behavior is too far from the familiar one.

Key facts. Cited by 11 papers in the map (data through 13 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 49 of its 63 references that are in the map. First posted on arXiv 21 May 2022.

Builds on

49 of 63 references.

Closest ancestors · 21

Same direction, newest first

  1. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning26 Apr
  2. 2022Comparing Model-free and Model-based Algorithms for Offline Reinforcement Learning14 Jan
  3. 2021RvS: What is Essential for Offline RL via Supervised Learning?20 Dec
  4. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  5. 2021Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters8 Oct
  6. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning8 Oct
  7. 2021Behavior Constraining in Weight Space for Offline Reinforcement Learning12 Jul
  8. 2021Offline RL Without Off-Policy Evaluation16 Jun
  9. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  10. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  11. 2021Risk-Averse Offline Reinforcement Learning10 Feb
  12. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning26 Oct
9 more
  1. 2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learning12 Aug
  2. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  3. 2020Hyperparameter Selection for Offline Reinforcement Learning17 Jul
  4. 2020Critic Regularized Regression26 Jun
  5. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  6. 2020MOPO: Model-based Offline Policy Optimization27 May
  7. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  8. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  9. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 28

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,531
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,231
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  5. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 767
  6. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 654
  7. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  8. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  9. 2019Human-Centered Tools for Coping with Imperfect Algorithms during Medical Decision-Makingcited by 109
  10. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
  11. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  12. 2020GradientDICE: Rethinking Generalized Offline Estimation of Stationary Valuescited by 70
16 more
  1. 2021On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learningcited by 59
  2. 2021Offline Meta-Reinforcement Learning with Online Self-Supervisioncited by 52
  3. 2016Learning and Policy Search in Stochastic Dynamical Systems with Bayesian Neural Networkscited by 49
  4. 2017Interpretable Policies for Reinforcement Learning by Genetic Programmingcited by 47
  5. 2022On the Utility of Prediction Sets in Human-AI Teamscited by 34
  6. 2021Autoregressive Dynamics Models for Offline Policy Evaluation and Optimizationcited by 33
  7. 2017A Benchmark Environment Motivated by Industrial Control Problemscited by 31
  8. 2021Active Offline Policy Selectioncited by 26
  9. 2020Clinician-in-the-Loop Decision Making: Reinforcement Learning with Near-Optimal Set-Valued Policiescited by 24
  10. 2019Safe Policy Improvement with Soft Baseline Bootstrappingcited by 22
  11. 2021Leveraging Expert Consistency to Improve Algorithmic Decision Supportcited by 17
  12. 2021Controlling Neural Networks with Rule Representationscited by 16
  13. 2021Conservative and Adaptive Penalty for Model-Based Safe Reinforcement Learningcited by 13
  14. 2020Reducing Sampling Error in Batch Temporal Difference Learningcited by 11
  15. 2014Non-Deterministic Policies in Markovian Decision Processescited by 7
  16. 2017Batch Reinforcement Learning on the Industrial Benchmark: First Experiencescited by 5

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 4; 2024: 4; 2025: 1; 2026: 2.

11 papers.

  1. 2023Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learningcited by 26
  2. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 15
  3. 2023Value function estimation using conditional diffusion models for controlcited by 10
  4. 2023Learning Control Policies for Variable Objectives from Offline Datacited by 3
  5. 2024Debiased Offline Representation Learning for Fast Online Adaptation in Non-stationary Dynamicscited by 2
  6. 2024Preference-Optimized Pareto Set Learning for Blackbox Optimizationcited by 2
  7. 2026Model-based Offline RL via Robust Value-Aware Model Learning with Implicitly Differentiable Adaptive Weightingcited by 2
  8. 2025Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatismcited by 1
  9. 2023Automatic Trade-off Adaptation in Offline RLcited by 0
  10. 2024TEA: Trajectory Encoding Augmentation for Robust and Transferable Policies in Offline Reinforcement Learningcited by 0
  11. 2026From Static Policies to Adaptive Priors in Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Clean Slate for Offline Reinforcement Learning
  2. 2026Active Offline-to-Online Reinforcement Learning
  3. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  4. 2022Leveraging Offline Data in Online Reinforcement Learning
  5. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
  6. 2023Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Bias
  7. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
  8. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory