mlsanityData to 7 Oct 2026

Deploying Offline Reinforcement Learning with Human Feedback

arXiv 2303.07046 ↗First posted 13 Mar 2023cs.LG cs.AIcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Reinforcement learning (RL) has shown promise for decision-making tasks in real-world applications. One practical framework involves training parameterized policy models from an offline dataset and subsequently deploying them in an online environment. However, this approach can be risky since the offline training may not be perfect, leading to poor performance of the RL models that may take dangerous actions. To address this issue, we propose an alternative framework that involves a human supervising the RL models and providing additional feedback in the online deployment phase. We formalize this online deployment problem and develop two approaches. The first approach uses model selection and the upper confidence bound algorithm to adaptively select a model to deploy from a candidate set of trained offline RL models. The second approach involves fine-tuning the model in the online deployment phase when a supervision signal arrives. We demonstrate the effectiveness of these approaches for robot locomotion control and traffic light control tasks through empirical validation.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 53 references that are in the map. First posted on arXiv 13 Mar 2023.

Builds on

26 of 53 references.

References · 26

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  6. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  9. 2014How transferable are features in deep neural networks?cited by 1,216
  10. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  11. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  12. 2018Do Better ImageNet Models Transfer Better?cited by 560
14 more
  1. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  2. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  3. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
  4. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
  5. 2019Mastering Complex Control in MOBA Games with Deep Reinforcement Learningcited by 150
  6. 2019CityFlow: A Multi-Agent Reinforcement Learning Environment for Large Scale City Traffic Scenariocited by 114
  7. 2020Batch Value-function Approximation with Only Realizabilitycited by 103
  8. 2018Playing hard exploration games by watching YouTubecited by 98
  9. 2020Error Bounds of Imitating Policies and Environmentscited by 90
  10. 2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90
  11. 2018Fast Policy Learning through Imitation and Reinforcementcited by 26
  12. 2020Cost-Sensitive Portfolio Selection via Deep Reinforcement Learningcited by 21
  13. 2020Batch-Constrained Distributional Reinforcement Learning for Session-based Recommendationcited by 4
  14. 2021Value Penalized Q-Learning for Recommender Systemscited by 4

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024A Survey On Enhancing Reinforcement Learning in Complex Environments: Insights from Human and LLM Feedbackcited by 9
  2. 2025A Survey of Reinforcement Learning for Optimization in Automationcited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning
  2. 2023Finetuning Offline World Models in the Real World
  3. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies
  4. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
  5. 2020MOPO: Model-based Offline Policy Optimization
  6. 2025Video-Enhanced Offline Reinforcement Learning: A Model-Based Approach
  7. 2026Active Offline-to-Online Reinforcement Learning
  8. 2025A Clean Slate for Offline Reinforcement Learning