mlsanityData to 7 Oct 2026

Human-centric Dialog Training via Offline Reinforcement Learning

arXiv 2010.05848 ↗First posted 12 Oct 2020cs.CL cs.LGcited by 92

On the map: Offline RL and imitation learning from data → Language-conditioned policies and VLAs

Abstract (arXiv)

How can we train a dialog model to produce better conversations by learning from human feedback, without the risk of humans teaching it harmful chat behaviors? We start by hosting models online, and gather human feedback from real-time, open-ended conversations, which we then use to train and improve the models using offline reinforcement learning (RL). We identify implicit conversational cues including language similarity, elicitation of laughter, sentiment, and more, which indicate positive human feedback, and embed these in multiple reward functions. A well-known challenge is that learning an RL policy in an offline setting usually fails due to the lack of ability to explore and the tendency to make over-optimistic estimates of future reward. These problems become even harder when using RL for language models, which can easily have a 20,000 action vocabulary and many possible reward functions. We solve the challenge by developing a novel class of offline RL algorithms. These algorithms use KL-control to penalize divergence from a pre-trained prior language model, and use a new strategy to make the algorithm pessimistic, instead of optimistic, in the face of uncertainty. We test the resulting dialog model with ratings from 80 users in an open-domain setting and find it achieves significant improvements over existing deep offline RL approaches. The novel offline RL method is viable for improving any existing generative dialog model using a static dataset of human feedback.

Key facts. Cited by 92 papers in the map (data through 7 Oct 2026). Placed in Language-conditioned policies and VLAs, one of 13 directions in Offline RL and imitation learning from data. Builds on 43 of its 79 references that are in the map. First posted on arXiv 12 Oct 2020.

Builds on

43 of 79 references.

Closest ancestors · 1

Same direction, newest first

  1. 2018Dialogue Generation: From Imitation Learning to Inverse Reinforcement Learning9 Dec

Other references · 42

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  6. 2015Trust Region Policy Optimizationcited by 2,503
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  8. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  9. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  10. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  11. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  12. 2018Universal Sentence Encodercited by 485
30 more
  1. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  2. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  3. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  4. 2018Maximum a Posteriori Policy Optimisationcited by 318
  5. 2016Deep Reinforcement Learning for Dialogue Generationcited by 286
  6. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  7. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  8. 2015Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Modelscited by 207
  9. 2016A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialoguescited by 174
  10. 2018The Design and Implementation of XiaoIce, an Empathetic Social Chatbotcited by 172
  11. 2018More Robust Doubly Robust Off-policy Evaluationcited by 141
  12. 2019What makes a good conversation? How controllable attributes affect human judgmentscited by 140
  13. 2020Unsupervised Evaluation of Interactive Dialog with DialoGPTcited by 139
  14. 2017Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasmcited by 117
  15. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RLcited by 113
  16. 2017Adversarial Learning for Neural Dialogue Generationcited by 103
  17. 2019Learning from Dialogue after Deployment: Feed Yourself, Chatbot!cited by 102
  18. 2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 100
  19. 2018Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systemscited by 53
  20. 2019Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shiftcited by 47
  21. 2016Dialogue Learning With Human-In-The-Loopcited by 46
  22. 2019Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systemscited by 42
  23. 2019An Optimistic Perspective on Offline Reinforcement Learningcited by 37
  24. 2017A Deep Reinforcement Learning Chatbotcited by 36
  25. 2019Hierarchical Reinforcement Learning for Open-Domain Dialogcited by 27
  26. 2017Iterative Policy Learning in End-to-End Trainable Task-Oriented Neural Dialog Modelscited by 26
  27. 2017Sample-efficient Actor-Critic Reinforcement Learning with Supervised Data for Dialogue Managementcited by 18
  28. 2018A Hierarchical Latent Structure for Variational Conversation Modelingcited by 17
  29. 2018Sentiment Adaptive End-to-End Dialog Systemscited by 17
  30. 2016Policy Networks with Two-Stage Training for Dialogue Systemscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 13; 2023: 23; 2024: 14; 2025: 17; 2026: 11.

92 papers.

Sort:

By area:

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  3. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  4. 2023Pretraining Language Models with Human Preferencescited by 245
  5. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  6. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  7. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 212
  8. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  9. 2021Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Surveycited by 130
  10. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 105
  11. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 86
  12. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedbackcited by 86
38 more
  1. 2023LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Modelscited by 84
  2. 2023Training Socially Aligned Language Models on Simulated Social Interactionscited by 77
  3. 2022Can Wikipedia Help Offline Reinforcement Learning?cited by 76
  4. 2023Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimismcited by 69
  5. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learningcited by 64
  6. 2024A Dense Reward View on Aligning Text-to-Image Diffusion with Preferencecited by 55
  7. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Valuescited by 53
  8. 2021Modeling Strong and Human-Like Gameplay with KL-Regularized Searchcited by 50
  9. 2021Efficient (Soft) Q-Learning for Text Generation with Limited Good Datacited by 35
  10. 2024DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory Stitchingcited by 35
  11. 2022Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizabilitycited by 32
  12. 2024SuperCorrect: Advancing Small LLM Reasoning with Thought Template Distillation and Self-Correctioncited by 32
  13. 2023Preference-grounded Token-level Guidance for Language Model Fine-tuningcited by 31
  14. 2023Ask an Expert: Leveraging Language Models to Improve Strategic Reasoning in Goal-Oriented Dialogue Modelscited by 30
  15. 2021Offline Reinforcement Learning with Soft Behavior Regularizationcited by 28
  16. 2023Aligning Language Models with Human Preferences via a Bayesian Approachcited by 27
  17. 2023On Diversified Preferences of Large Language Model Alignmentcited by 26
  18. 2023The Generalization Gap in Offline Reinforcement Learningcited by 25
  19. 2022Let Offline RL Flow: Training Conservative Agents in the Latent Space of Normalizing Flowscited by 21
  20. 2024Advancing Social Intelligence in AI Agents: Technical Challenges and Open Questionscited by 21
  21. 2021Continual Learning for Grounded Instruction Generation by Observing Human Following Behaviorcited by 19
  22. 2023Fantastic Rewards and How to Tame Them: A Case Study on Reward Learning for Task-oriented Dialogue Systemscited by 19
  23. 2024Prompt-Based Length Controlled Generation with Multiple Control Typescited by 19
  24. 2024Sample-Efficient Alignment for LLMscited by 16
  25. 2024SILMM: Self-Improving Large Multimodal Models for Compositional Text-to-Image Generationcited by 15
  26. 2022Simulating Bandit Learning from User Feedback for Extractive Question Answeringcited by 14
  27. 2023Prompt-Based Length Controlled Generation with Reinforcement Learningcited by 14
  28. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  29. 2023Building Persona Consistent Dialogue Agents with Offline Reinforcement Learningcited by 13
  30. 2025The Era of Real-World Human Interaction: RL from User Conversationscited by 13
  31. 2022Knowledge Infused Decodingcited by 12
  32. 2025Segmenting Text and Learning Their Rewards for Improved RLHF in Language Modelcited by 12
  33. 2021Braxlines: Fast and Interactive Toolkit for RL-driven Behavior Engineering beyond Reward Maximizationcited by 11
  34. 2022Towards Boosting the Open-Domain Chatbot with Human Feedbackcited by 11
  35. 2023Continually Improving Extractive QA via Human Feedbackcited by 11
  36. 2024Problem Solving Through Human-AI Preference-Based Cooperationcited by 11
  37. 2023Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challengescited by 10
  38. 2025Offline RL by Reward-Weighted Fine-Tuning for Conversation Optimizationcited by 10

50 most cited of 92 All 92 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022CHAI: A CHatbot AI for Task-Oriented Dialogue with Offline Reinforcement Learning
  2. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
  3. 2025SpeakRL: Synergizing Reasoning, Speaking, and Acting in Language Models with Reinforcement Learning
  4. 2021Integrating Pretrained Language Model for Dialogue Policy Learning
  5. 2022A Mixture-of-Expert Approach to RL-based Dialogue Management
  6. 2024Dual Active Learning for Reinforcement Learning from Human Feedback
  7. 2023Offline Reinforcement Learning for Mixture-of-Expert Dialogue Management
  8. 2020Guided Dialog Policy Learning without Adversarial Learning in the Loop