# Adversarial Learning of Task-Oriented Neural Dialog Models

arXiv: https://arxiv.org/abs/1805.11762 · HTML: https://mlsanity.com/paper/1805.11762/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 May 2018.

## Built on by (most cited first, 17 of 17)

- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [AI Safety in Generative AI Large Language Models: A Survey](https://mlsanity.com/paper/2407.18369/) (2024)
- [A Survey on Recent Advances and Challenges in Reinforcement Learning Methods for Task-Oriented Dialogue Policy Learning](https://mlsanity.com/paper/2202.13675/) (2022)
- [Interactive Grounded Language Understanding in a Collaborative Environment: IGLU 2021](https://mlsanity.com/paper/2205.02388/) (2022)
- [A Survey of the Evolution of Language Model-Based Dialogue Systems: Data, Task and Models](https://mlsanity.com/paper/2311.16789/) (2023)
- [NeurIPS 2021 Competition IGLU: Interactive Grounded Language Understanding in a Collaborative Environment](https://mlsanity.com/paper/2110.06536/) (2021)
- [IGLU 2022: Interactive Grounded Language Understanding in a Collaborative Environment at NeurIPS 2022](https://mlsanity.com/paper/2205.13771/) (2022)
- [Transforming Human-Centered AI Collaboration: Redefining Embodied Agents Capabilities through Interactive Grounded Language Instructions](https://mlsanity.com/paper/2305.10783/) (2023)
- [DEUS: A Data-driven Approach to Estimate User Satisfaction in Multi-turn Dialogues](https://mlsanity.com/paper/2103.01287/) (2021)
- [Imperfect also Deserves Reward: Multi-Level and Sequential Reward Modeling for Better Dialog Management](https://mlsanity.com/paper/2104.04748/) (2021)
- [Saliency Attention and Semantic Similarity-Driven Adversarial Perturbation](https://mlsanity.com/paper/2406.19413/) (2024)
- ["Think Before You Speak": Improving Multi-Action Dialog Policy by Planning Single-Action Dialogs](https://mlsanity.com/paper/2204.11481/) (2022)
- [Adversarial learning of neural user simulators for dialogue policy optimisation](https://mlsanity.com/paper/2306.00858/) (2023)
- [IDAT: A Multi-Modal Dataset and Toolkit for Building and Evaluating Interactive Task-Solving Agents](https://mlsanity.com/paper/2407.08898/) (2024)
- [Two Approaches to Building Collaborative, Task-Oriented Dialog Agents through Self-Play](https://mlsanity.com/paper/2109.09597/) (2021)
- [Multi-Action Dialog Policy Learning from Logged User Feedback](https://mlsanity.com/paper/2302.13505/) (2023)
- [Why Guided Dialog Policy Learning performs well? Understanding the role of adversarial learning and its alternative](https://mlsanity.com/paper/2307.06721/) (2023)

## Abstract (arXiv)

In this work, we propose an adversarial learning method for reward estimation in reinforcement learning (RL) based task-oriented dialog models. Most of the current RL based task-oriented dialog systems require the access to a reward signal from either user feedback or user ratings. Such user ratings, however, may not always be consistent or available in practice. Furthermore, online dialog policy learning with RL typically requires a large number of queries to users, suffering from sample efficiency problem. To address these challenges, we propose an adversarial learning method to learn dialog rewards directly from dialog samples. Such rewards are further used to optimize the dialog policy with policy gradient based RL. In the evaluation in a restaurant search domain, we show that the proposed adversarial dialog learning method achieves advanced dialog success rate comparing to strong baseline methods. We further discuss the covariate shift problem in online adversarial dialog learning and show how we can address that with partial access to user feedback.