Data to 7 Oct 2026

Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog

arXiv 1907.00456 ↗First posted 30 Jun 2019cs.LG cs.AI stat.MLcited by 283

On the map: Offline RL and imitation learning from data → Language-conditioned policies and VLAs

Abstract (arXiv)

Most deep reinforcement learning (RL) systems are not able to learn effectively from off-policy data, especially if they cannot explore online in the environment. These are critical shortcomings for applying RL to real-world problems where collecting data is expensive, and models must be tested offline before being deployed to interact with the environment -- e.g. systems that learn from human interaction. Thus, we develop a novel class of off-policy batch RL algorithms, which are able to effectively learn offline, without exploring, from a fixed batch of human interaction data. We leverage models pre-trained on data as a strong prior, and use KL-control to penalize divergence from this prior during RL training. We also use dropout-based uncertainty estimates to lower bound the target Q-values as a more efficient alternative to Double Q-Learning. The algorithms are tested on the problem of open-domain dialog generation -- a challenging reinforcement learning problem with a 20,000-dimensional action space. Using our Way Off-Policy algorithm, we can extract multiple different reward functions post-hoc from collected human interaction data, and learn effectively from all of these. We test the real-world generalization of these systems by deploying them live to converse with humans in an open-domain setting, and demonstrate that our algorithm achieves significant improvements over prior methods in off-policy batch RL.

Key facts. Cited by 283 papers in the map (data through 7 Oct 2026). Placed in Language-conditioned policies and VLAs, one of 13 directions in Offline RL and imitation learning from data. Builds on 42 of its 72 references that are in the map. First posted on arXiv 30 Jun 2019.

Builds on

42 of 72 references.

Closest ancestors · 1

Same direction, newest first

2018Dialogue Generation: From Imitation Learning to Inverse Reinforcement Learning9 Dec

Other references · 41

2015Distilling the Knowledge in a Neural Networkcited by 10,076
2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
2017Deep reinforcement learning from human preferencescited by 3,846
2013Playing Atari with Deep Reinforcement Learningcited by 3,211
2015Trust Region Policy Optimizationcited by 2,503
2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
2017Hindsight Experience Replaycited by 1,096
2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
2017Reinforcement Learning with Deep Energy-Based Policiescited by 659

29 more

2016Deep Exploration via Bootstrapped DQNcited by 518
2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
2018Maximum a Posteriori Policy Optimisationcited by 318
2016Deep Reinforcement Learning for Dialogue Generationcited by 286
2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
2015Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Modelscited by 207
2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 178
2016A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialoguescited by 174
2018The Design and Implementation of XiaoIce, an Empathetic Social Chatbotcited by 172
2018More Robust Doubly Robust Off-policy Evaluationcited by 141
2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 125
2017Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasmcited by 117
2017Adversarial Learning for Neural Dialogue Generationcited by 103
2019Learning from Dialogue after Deployment: Feed Yourself, Chatbot!cited by 102
2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 100
2017Uncertainty-Aware Reinforcement Learning for Collision Avoidancecited by 86
2018Efficient Exploration through Bayesian Deep Q-Networkscited by 67
2018Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systemscited by 53
2019Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shiftcited by 47
2016Dialogue Learning With Human-In-The-Loopcited by 46
2019Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systemscited by 42
2017A Deep Reinforcement Learning Chatbotcited by 36
2017Iterative Policy Learning in End-to-End Trainable Task-Oriented Neural Dialog Modelscited by 26
2017Sample-efficient Actor-Critic Reinforcement Learning with Supervised Data for Dialogue Managementcited by 18
2018A Hierarchical Latent Structure for Variational Conversation Modelingcited by 17
2018Sentiment Adaptive End-to-End Dialog Systemscited by 17
2016Policy Networks with Two-Stage Training for Dialogue Systemscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 6; 2020: 16; 2021: 41; 2022: 58; 2023: 50; 2024: 54; 2025: 36; 2026: 22.

283 papers.

Sort:

By area:

2022Training language models to follow instructions with human feedbackcited by 15,712
2020Learning to summarize from human feedbackcited by 2,562
2019Fine-Tuning Language Models from Human Preferencescited by 2,124
2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
2022Red Teaming Language Models with Language Modelscited by 983
2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
2019Behavior Regularized Offline Reinforcement Learningcited by 654
2020MOPO: Model-based Offline Policy Optimizationcited by 630
2020MOReL : Model-Based Offline Reinforcement Learningcited by 582

38 more

2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
2020Is Pessimism Provably Efficient for Offline RL?cited by 362
2022Teaching language models to support answers with verified quotescited by 290
2021Recursively Summarizing Books with Human Feedbackcited by 287
2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
2023Nash Learning from Human Feedbackcited by 220
2022Optimizing Prompts for Text-to-Image Generationcited by 204
2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 164
2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
2021Offline RL Without Off-Policy Evaluationcited by 150
2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 136
2020PLAS: Latent Action Space for Offline Reinforcement Learningcited by 133
2022RL with KL penalties is better viewed as Bayesian inferencecited by 130
2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 129
2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 125
2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 118
2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
2020Hyperparameter Selection for Offline Reinforcement Learningcited by 113
2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 109
2019Benchmarking Batch Deep Reinforcement Learning Algorithmscited by 108
2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 108
2020Offline Reinforcement Learning from Images with Latent Space Modelscited by 94
2020Offline Meta-Reinforcement Learning with Advantage Weightingcited by 92
2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
2021Representation Matters: Offline Pretraining for Sequential Decision Makingcited by 90
2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 90
2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
2025SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learningcited by 89
2022A data-driven approach for learning to control computerscited by 87
2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningcited by 85
2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 80

50 most cited of 283 All 283 (JSON)

Similar, not cited

Close in wording, no citation either way