mlsanityData to 7 Oct 2026

CHAI: A CHatbot AI for Task-Oriented Dialogue with Offline Reinforcement Learning

arXiv 2204.08426 ↗First posted 18 Apr 2022cs.CL cs.LGcited by 33

On the map: Offline RL and imitation learning from data → Language-conditioned policies and VLAs

Abstract (arXiv)

Conventionally, generation of natural language for dialogue agents may be viewed as a statistical learning problem: determine the patterns in human-provided data and generate appropriate responses with similar statistical properties. However, dialogue can also be regarded as a goal directed process, where speakers attempt to accomplish a specific task. Reinforcement learning (RL) algorithms are designed specifically for solving such goal-directed problems, but the most direct way to apply RL -- through trial-and-error learning in human conversations, -- is costly. In this paper, we study how offline reinforcement learning can instead be used to train dialogue agents entirely using static datasets collected from human speakers. Our experiments show that recently developed offline RL methods can be combined with language models to yield realistic dialogue agents that better accomplish task goals.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Placed in Language-conditioned policies and VLAs, one of 13 directions in Offline RL and imitation learning from data. Builds on 35 of its 58 references that are in the map. First posted on arXiv 18 Apr 2022.

Builds on

35 of 58 references.

Closest ancestors · 1

Same direction, newest first

  1. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog30 Jun

Other references · 34

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  6. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  8. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  9. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  10. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  11. 2020A Simple Language Model for Task-Oriented Dialoguecited by 331
  12. 2016Deep Reinforcement Learning for Dialogue Generationcited by 286
22 more
  1. 2017Deal or No Deal? End-to-End Learning for Negotiation Dialoguescited by 202
  2. 2018Neural Approaches to Conversational AIcited by 164
  3. 2018Decoupling Strategy and Generation in Negotiation Dialoguescited by 156
  4. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RLcited by 113
  5. 2019Task-Oriented Dialog Systems that Consider Multiple Appropriate Responses under the Same Contextcited by 99
  6. 2020SOLOIST: Building Task Bots at Scale with Transfer Learning and Machine Teachingcited by 71
  7. 2018Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systemscited by 53
  8. 2019Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Modelscited by 53
  9. 2019Semantically Conditioned Dialog Response Generation via Hierarchical Disentangled Self-Attentioncited by 43
  10. 2016A Sequence-to-Sequence Model for User Simulation in Spoken Dialogue Systemscited by 39
  11. 2020Modelling Hierarchical Structure between Dialogue Policy and Natural Language Generator with Option Framework for Task-oriented Dialogue Systemcited by 28
  12. 2016Sequence-to-Sequence Generation for Spoken Dialogue via Deep Syntax Trees and Stringscited by 27
  13. 2019Hierarchical Reinforcement Learning for Open-Domain Dialogcited by 27
  14. 2019Alternating Recurrent Dialog Model with Large-scale Pre-trained Language Modelscited by 25
  15. 2017Hierarchical Text Generation and Planning for Strategic Dialoguecited by 23
  16. 2017Sample-efficient Actor-Critic Reinforcement Learning with Supervised Data for Dialogue Managementcited by 18
  17. 2016Multiresolution Recurrent Neural Networks: An Application to Dialogue Response Generationcited by 16
  18. 2016Policy Networks with Two-Stage Training for Dialogue Systemscited by 14
  19. 2017A Copy-Augmented Sequence-to-Sequence Architecture Gives Good Performance on Task-Oriented Dialoguecited by 14
  20. 2016Continuously Learning Neural Dialogue Managementcited by 12
  21. 2015Learning from Real Users: Rating Dialogue Success with Neural Networks for Reinforcement Learning in Spoken Dialogue Systemscited by 8
  22. 2016Coherent Dialogue with Attention-based Language Modelscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 13; 2024: 7; 2025: 6; 2026: 1.

33 papers.

Sort:

By area:

  1. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  2. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 212
  3. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  4. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  5. 2024Steering Your Generalists: Improving Robotic Foundation Models via Value Guidancecited by 100
  6. 2023LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Modelscited by 84
  7. 2023Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agentscited by 66
  8. 2023A Survey on Transformers in Reinforcement Learningcited by 50
  9. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
  10. 2025Digi-Q: Learning Q-Value Functions for Training Device-Control Agentscited by 20
  11. 2023Fantastic Rewards and How to Tame Them: A Case Study on Reward Learning for Task-oriented Dialogue Systemscited by 19
  12. 2023Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Modelscited by 17
21 more
  1. 2022A Mixture-of-Expert Approach to RL-based Dialogue Managementcited by 16
  2. 2024VerifierQ: Enhancing LLM Test Time Compute with Q-Learning-based Verifierscited by 14
  3. 2023Building Persona Consistent Dialogue Agents with Offline Reinforcement Learningcited by 13
  4. 2025Reinforcement Learning for Machine Learning Engineering Agentscited by 13
  5. 2023Towards the Scalable Evaluation of Cooperativeness in Language Modelscited by 12
  6. 2024Dialogue Action Tokens: Steering Language Models in Goal-Directed Dialogue with a Multi-Turn Plannercited by 12
  7. 2022KRLS: Improving End-to-End Response Generation in Task Oriented Dialog with Reinforced Keywords Learningcited by 9
  8. 2024TrajDeleter: Enabling Trajectory Forgetting in Offline Reinforcement Learning Agentscited by 7
  9. 2023Offline Reinforcement Learning with Imbalanced Datasetscited by 6
  10. 2023On the Effectiveness of Offline RL for Dialogue Response Generationcited by 6
  11. 2025ADAPT: Actively Discovering and Adapting to Preferences for any Taskcited by 5
  12. 2025Surfer 2: The Next Generation of Cross-Platform Computer Use Agentscited by 5
  13. 2022Dialogue Evaluation with Offline Reinforcement Learningcited by 3
  14. 2023Offline Reinforcement Learning for Mixture-of-Expert Dialogue Managementcited by 3
  15. 2025Iterative GRPO: Batch-Online Multi-Turn RL via Single-Turn RLHFcited by 2
  16. 2022CausalDialogue: Modeling Utterance-level Causality in Conversationscited by 1
  17. 2023Deep RL with Hierarchical Action Exploration for Dialogue Generationcited by 1
  18. 2024Towards Goal-Oriented Agents for Evolving Problems Observed via Conversationcited by 1
  19. 2025Prompt Optimization with Logged Bandit Datacited by 1
  20. 2024Plan of Thoughts: Heuristic-Guided Problem Solving with Large Language Modelscited by 0
  21. 2026From Static Context to Calibrated Interactive RL: Mitigating Distribution Shift in Multi-turn Dialogue with Aligned Simulatorcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020Human-centric Dialog Training via Offline Reinforcement Learning
  2. 2024Interactive Dialogue Agents via Reinforcement Learning on Hindsight Regenerations
  3. 2025SpeakRL: Synergizing Reasoning, Speaking, and Acting in Language Models with Reinforcement Learning
  4. 2020Rethinking Supervised Learning and Reinforcement Learning in Task-Oriented Dialogue Systems
  5. 2024Improving Multi-Domain Task-Oriented Dialogue System with Offline Reinforcement Learning
  6. 2024Goal Inference from Open-Ended Dialog
  7. 2022Toward Self-learning End-to-End Task-Oriented Dialog Systems
  8. 2023Dialogue Agents 101: A Beginner's Guide to Critical Ingredients for Designing Effective Conversational Systems