# Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systems

arXiv: https://arxiv.org/abs/1804.06512 · HTML: https://mlsanity.com/paper/1804.06512/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Apr 2018.

## Built on by (most cited first, 30 of 53)

- [Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog](https://mlsanity.com/paper/1907.00456/) (2019)
- [Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization](https://mlsanity.com/paper/2406.01171/) (2024)
- [DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback](https://mlsanity.com/paper/2311.10081/) (2023)
- [GALAXY: A Generative Pre-trained Model for Task-Oriented Dialog with Semi-Supervised Learning and Explicit Policy Injection](https://mlsanity.com/paper/2111.14592/) (2021)
- [Human-centric Dialog Training via Offline Reinforcement Learning](https://mlsanity.com/paper/2010.05848/) (2020)
- [RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs](https://mlsanity.com/paper/2404.08555/) (2024)
- [Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback](https://mlsanity.com/paper/2303.05453/) (2023)
- [Prompt-Based Monte-Carlo Tree Search for Goal-Oriented Dialogue Policy Planning](https://mlsanity.com/paper/2305.13660/) (2023)
- [Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation](https://mlsanity.com/paper/2305.00955/) (2023)
- [The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values](https://mlsanity.com/paper/2310.07629/) (2023)
- [Putting Humans in the Natural Language Processing Loop: A Survey](https://mlsanity.com/paper/2103.04044/) (2021)
- [CHAI: A CHatbot AI for Task-Oriented Dialogue with Offline Reinforcement Learning](https://mlsanity.com/paper/2204.08426/) (2022)
- [Planning Like Human: A Dual-process Framework for Dialogue Planning](https://mlsanity.com/paper/2406.05374/) (2024)
- [TIAGE: A Benchmark for Topic-Shift Aware Dialog Modeling](https://mlsanity.com/paper/2109.04562/) (2021)
- [Context-Aware Language Modeling for Goal-Oriented Dialogue Systems](https://mlsanity.com/paper/2204.10198/) (2022)
- [Continual Learning for Instruction Following from Realtime Feedback](https://mlsanity.com/paper/2212.09710/) (2022)
- [Continual Learning for Grounded Instruction Generation by Observing Human Following Behavior](https://mlsanity.com/paper/2108.04812/) (2021)
- [Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2104.08570/) (2021)
- [Interacting with Non-Cooperative User: A New Paradigm for Proactive Dialogue Policy](https://mlsanity.com/paper/2204.07433/) (2022)
- [End-to-end Task-oriented Dialogue: A Survey of Tasks, Methods, and Future Directions](https://mlsanity.com/paper/2311.09008/) (2023)
- [Multimodal Conversational AI: A Survey of Datasets and Approaches](https://mlsanity.com/paper/2205.06907/) (2022)
- [KETOD: Knowledge-Enriched Task-Oriented Dialogue](https://mlsanity.com/paper/2205.05589/) (2022)
- [Large Language Models as User-Agents for Evaluating Task-Oriented-Dialogue Systems](https://mlsanity.com/paper/2411.09972/) (2024)
- [Preemptive Detection and Correction of Misaligned Actions in LLM Agents](https://mlsanity.com/paper/2407.11843/) (2024)
- [Toward Self-learning End-to-End Task-Oriented Dialog Systems](https://mlsanity.com/paper/2201.06849/) (2022)
- [Dealing with Semantic Underspecification in Multimodal NLP](https://mlsanity.com/paper/2306.05240/) (2023)
- [Dynamic Planning in Open-Ended Dialogue using Reinforcement Learning](https://mlsanity.com/paper/2208.02294/) (2022)
- [Multi-Granularity Prompts for Topic Shift Detection in Dialogue](https://mlsanity.com/paper/2305.14006/) (2023)
- [SYNDICOM: Improving Conversational Commonsense with Error-Injection and Natural Language Feedback](https://mlsanity.com/paper/2309.10015/) (2023)
- [Enhancing End-to-End Multi-Task Dialogue Systems: A Study on Intrinsic Motivation Reinforcement Learning Algorithms for Improved Training and Adaptability](https://mlsanity.com/paper/2401.18040/) (2024)

## Abstract (arXiv)

In this work, we present a hybrid learning method for training task-oriented dialogue systems through online user interactions. Popular methods for learning task-oriented dialogues include applying reinforcement learning with user feedback on supervised pre-training models. Efficiency of such learning method may suffer from the mismatch of dialogue state distribution between offline training and online interactive learning stages. To address this challenge, we propose a hybrid imitation and reinforcement learning method, with which a dialogue agent can effectively learn from its interaction with users by learning from human teaching and feedback. We design a neural network based task-oriented dialogue agent that can be optimized end-to-end with the proposed learning method. Experimental results show that our end-to-end dialogue agent can learn effectively from the mistake it makes via imitation learning from user teaching. Applying reinforcement learning with user feedback after the imitation learning stage further improves the agent's capability in successfully completing a task.