# CHAI: A CHatbot AI for Task-Oriented Dialogue with Offline Reinforcement Learning

arXiv: https://arxiv.org/abs/2204.08426 · HTML: https://mlsanity.com/paper/2204.08426/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Placed in Language-conditioned policies and VLAs, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 35 of its 58 references that are in the map.
- First posted on arXiv 18 Apr 2022.

## Placed in

- Offline RL and imitation learning from data / [Language-conditioned policies and VLAs](https://mlsanity.com/offlinerl/language-conditioned-policies-and-vlas/)

## Builds on (most cited first, 30 of 35)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction](https://mlsanity.com/paper/1906.00949/) (2019)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [Behavior Regularized Offline Reinforcement Learning](https://mlsanity.com/paper/1911.11361/) (2019)
- [Towards a Human-like Open-Domain Chatbot](https://mlsanity.com/paper/2001.09977/) (2020)
- [A Simple Language Model for Task-Oriented Dialogue](https://mlsanity.com/paper/2005.00796/) (2020)
- [Deep Reinforcement Learning for Dialogue Generation](https://mlsanity.com/paper/1606.01541/) (2016)
- [Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog](https://mlsanity.com/paper/1907.00456/) (2019)
- [Deal or No Deal? End-to-End Learning for Negotiation Dialogues](https://mlsanity.com/paper/1706.05125/) (2017)
- [Neural Approaches to Conversational AI](https://mlsanity.com/paper/1809.08267/) (2018)
- [Decoupling Strategy and Generation in Negotiation Dialogues](https://mlsanity.com/paper/1808.09637/) (2018)
- [EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL](https://mlsanity.com/paper/2007.11091/) (2020)
- [Task-Oriented Dialog Systems that Consider Multiple Appropriate Responses under the Same Context](https://mlsanity.com/paper/1911.10484/) (2019)
- [SOLOIST: Building Task Bots at Scale with Transfer Learning and Machine Teaching](https://mlsanity.com/paper/2005.05298/) (2020)
- [Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1804.06512/) (2018)
- [Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models](https://mlsanity.com/paper/1902.08858/) (2019)
- [Semantically Conditioned Dialog Response Generation via Hierarchical Disentangled Self-Attention](https://mlsanity.com/paper/1905.12866/) (2019)
- [A Sequence-to-Sequence Model for User Simulation in Spoken Dialogue Systems](https://mlsanity.com/paper/1607.00070/) (2016)
- [Modelling Hierarchical Structure between Dialogue Policy and Natural Language Generator with Option Framework for Task-oriented Dialogue System](https://mlsanity.com/paper/2006.06814/) (2020)
- [Sequence-to-Sequence Generation for Spoken Dialogue via Deep Syntax Trees and Strings](https://mlsanity.com/paper/1606.05491/) (2016)
- [Hierarchical Reinforcement Learning for Open-Domain Dialog](https://mlsanity.com/paper/1909.07547/) (2019)
- [Alternating Recurrent Dialog Model with Large-scale Pre-trained Language Models](https://mlsanity.com/paper/1910.03756/) (2019)
- [Hierarchical Text Generation and Planning for Strategic Dialogue](https://mlsanity.com/paper/1712.05846/) (2017)
- [Sample-efficient Actor-Critic Reinforcement Learning with Supervised Data for Dialogue Management](https://mlsanity.com/paper/1707.00130/) (2017)
- [Multiresolution Recurrent Neural Networks: An Application to Dialogue Response Generation](https://mlsanity.com/paper/1606.00776/) (2016)

## Built on by (most cited first, 30 of 34)

- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL](https://mlsanity.com/paper/2402.19446/) (2024)
- [Foundation Models for Decision Making: Problems, Methods, and Opportunities](https://mlsanity.com/paper/2303.04129/) (2023)
- [Offline RL for Natural Language Generation with Implicit Language Q Learning](https://mlsanity.com/paper/2206.11871/) (2022)
- [Steering Your Generalists: Improving Robotic Foundation Models via Value Guidance](https://mlsanity.com/paper/2410.13816/) (2024)
- [LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models](https://mlsanity.com/paper/2311.18232/) (2023)
- [Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agents](https://mlsanity.com/paper/2303.00855/) (2023)
- [A Survey on Transformers in Reinforcement Learning](https://mlsanity.com/paper/2301.03044/) (2023)
- [On Transforming Reinforcement Learning by Transformer: The Development Trajectory](https://mlsanity.com/paper/2212.14164/) (2022)
- [Digi-Q: Learning Q-Value Functions for Training Device-Control Agents](https://mlsanity.com/paper/2502.15760/) (2025)
- [Fantastic Rewards and How to Tame Them: A Case Study on Reward Learning for Task-oriented Dialogue Systems](https://mlsanity.com/paper/2302.10342/) (2023)
- [Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models](https://mlsanity.com/paper/2305.14718/) (2023)
- [A Mixture-of-Expert Approach to RL-based Dialogue Management](https://mlsanity.com/paper/2206.00059/) (2022)
- [Building Persona Consistent Dialogue Agents with Offline Reinforcement Learning](https://mlsanity.com/paper/2310.10735/) (2023)
- [VerifierQ: Enhancing LLM Test Time Compute with Q-Learning-based Verifiers](https://mlsanity.com/paper/2410.08048/) (2024)
- [Reinforcement Learning for Machine Learning Engineering Agents](https://mlsanity.com/paper/2509.01684/) (2025)
- [Dialogue Action Tokens: Steering Language Models in Goal-Directed Dialogue with a Multi-Turn Planner](https://mlsanity.com/paper/2406.11978/) (2024)
- [Towards the Scalable Evaluation of Cooperativeness in Language Models](https://mlsanity.com/paper/2303.13360/) (2023)
- [KRLS: Improving End-to-End Response Generation in Task Oriented Dialog with Reinforced Keywords Learning](https://mlsanity.com/paper/2211.16773/) (2022)
- [TrajDeleter: Enabling Trajectory Forgetting in Offline Reinforcement Learning Agents](https://mlsanity.com/paper/2404.12530/) (2024)
- [Offline Reinforcement Learning with Imbalanced Datasets](https://mlsanity.com/paper/2307.02752/) (2023)
- [On the Effectiveness of Offline RL for Dialogue Response Generation](https://mlsanity.com/paper/2307.12425/) (2023)
- [ADAPT: Actively Discovering and Adapting to Preferences for any Task](https://mlsanity.com/paper/2504.04040/) (2025)
- [Surfer 2: The Next Generation of Cross-Platform Computer Use Agents](https://mlsanity.com/paper/2510.19949/) (2025)
- [Training Language Models for Bilateral Trade with Private Information](https://mlsanity.com/paper/2604.16472/) (2026)
- [Dialogue Evaluation with Offline Reinforcement Learning](https://mlsanity.com/paper/2209.00876/) (2022)
- [Offline Reinforcement Learning for Mixture-of-Expert Dialogue Management](https://mlsanity.com/paper/2302.10850/) (2023)
- [Iterative GRPO: Batch-Online Multi-Turn RL via Single-Turn RLHF](https://mlsanity.com/paper/2511.21638/) (2025)
- [CausalDialogue: Modeling Utterance-level Causality in Conversations](https://mlsanity.com/paper/2212.10515/) (2022)
- [Deep RL with Hierarchical Action Exploration for Dialogue Generation](https://mlsanity.com/paper/2303.13465/) (2023)

## Abstract (arXiv)

Conventionally, generation of natural language for dialogue agents may be viewed as a statistical learning problem: determine the patterns in human-provided data and generate appropriate responses with similar statistical properties. However, dialogue can also be regarded as a goal directed process, where speakers attempt to accomplish a specific task. Reinforcement learning (RL) algorithms are designed specifically for solving such goal-directed problems, but the most direct way to apply RL -- through trial-and-error learning in human conversations, -- is costly. In this paper, we study how offline reinforcement learning can instead be used to train dialogue agents entirely using static datasets collected from human speakers. Our experiments show that recently developed offline RL methods can be combined with language models to yield realistic dialogue agents that better accomplish task goals.