# Generating Challenge Datasets for Task-Oriented Conversational Agents through Self-Play

arXiv: https://arxiv.org/abs/1910.07357 · HTML: https://mlsanity.com/paper/1910.07357/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Oct 2019.

## Built on by (most cited first, 2 of 2)

- [Advising Agent for Service-Providing Live-Chat Operators](https://mlsanity.com/paper/2105.03986/) (2021)
- [Two Approaches to Building Collaborative, Task-Oriented Dialog Agents through Self-Play](https://mlsanity.com/paper/2109.09597/) (2021)

## Abstract (arXiv)

End-to-end neural approaches are becoming increasingly common in conversational scenarios due to their promising performances when provided with sufficient amount of data. In this paper, we present a novel methodology to address the interpretability of neural approaches in such scenarios by creating challenge datasets using dialogue self-play over multiple tasks/intents. Dialogue self-play allows generating large amount of synthetic data; by taking advantage of the complete control over the generation process, we show how neural approaches can be evaluated in terms of unseen dialogue patterns. We propose several out-of-pattern test cases each of which introduces a natural and unexpected user utterance phenomenon. As a proof of concept, we built a single and a multiple memory network, and show that these two architectures have diverse performances depending on the peculiar dialogue patterns.