# Rescue Conversations from Dead-ends: Efficient Exploration for Task-oriented Dialogue Policy Optimization

arXiv: https://arxiv.org/abs/2305.03262 · HTML: https://mlsanity.com/paper/2305.03262/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 68 references that are in the map.
- First posted on arXiv 5 May 2023.

## Builds on (most cited first, 30 of 33)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [Deep Exploration via Bootstrapped DQN](https://mlsanity.com/paper/1602.04621/) (2016)
- [Deal or No Deal? End-to-End Learning for Negotiation Dialogues](https://mlsanity.com/paper/1706.05125/) (2017)
- [A Survey on Dialogue Systems: Recent Advances and New Frontiers](https://mlsanity.com/paper/1711.01731/) (2017)
- [Interactive Path Reasoning on Graph for Conversational Recommendation](https://mlsanity.com/paper/2007.00194/) (2020)
- [A User Simulator for Task-Completion Dialogues](https://mlsanity.com/paper/1612.05688/) (2016)
- [Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models](https://mlsanity.com/paper/1902.08858/) (2019)
- [Unified Conversational Recommendation Policy Learning via Graph-based Reinforcement Learning](https://mlsanity.com/paper/2105.09710/) (2021)
- [Microsoft Dialogue Challenge: Building End-to-End Task-Completion Dialogue Systems](https://mlsanity.com/paper/1807.11125/) (2018)
- [Off-Policy Evaluation via Off-Policy Classification](https://mlsanity.com/paper/1906.01624/) (2019)
- [BBQ-Networks: Efficient Exploration in Deep Reinforcement Learning for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1711.05715/) (2017)
- [A Survey on Recent Advances and Challenges in Reinforcement Learning Methods for Task-Oriented Dialogue Policy Learning](https://mlsanity.com/paper/2202.13675/) (2022)
- [On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systems](https://mlsanity.com/paper/1605.07669/) (2016)
- [Multi-Agent Task-Oriented Dialog Policy Learning with Role-Aware Reward Decomposition](https://mlsanity.com/paper/2004.03809/) (2020)
- [Open-domain clarification question generation without question examples](https://mlsanity.com/paper/2110.09779/) (2021)
- [Policy Networks with Two-Stage Training for Dialogue Systems](https://mlsanity.com/paper/1606.03152/) (2016)
- [Budgeted Policy Learning for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1906.00499/) (2019)
- [Conversation as Action Under Uncertainty](https://mlsanity.com/paper/1301.3883/) (2013)
- [Adversarial Advantage Actor-Critic Model for Task-Completion Dialogue Policy Learning](https://mlsanity.com/paper/1710.11277/) (2017)
- [Subgoal Discovery for Hierarchical Dialogue Policy Learning](https://mlsanity.com/paper/1804.07855/) (2018)
- [Dialog Policy Learning for Joint Clarification and Active Learning Queries](https://mlsanity.com/paper/2006.05456/) (2020)
- [Goal-oriented Dialogue Policy Learning from Failures](https://mlsanity.com/paper/1808.06497/) (2018)
- [Retrieve & Memorize: Dialog Policy Learning with Multi-Action Memory](https://mlsanity.com/paper/2106.02317/) (2021)

## Built on by (most cited first, 1 of 1)

- [Dialogue Telemetry: Turn-Level Instrumentation for Autonomous Information Gathering](https://mlsanity.com/paper/2601.09570/) (2026)

## Abstract (arXiv)

Training a dialogue policy using deep reinforcement learning requires a lot of exploration of the environment. The amount of wasted invalid exploration makes their learning inefficient. In this paper, we find and define an important reason for the invalid exploration: dead-ends. When a conversation enters a dead-end state, regardless of the actions taken afterward, it will continue in a dead-end trajectory until the agent reaches a termination state or maximum turn. We propose a dead-end resurrection (DDR) algorithm that detects the initial dead-end state in a timely and efficient manner and provides a rescue action to guide and correct the exploration direction. To prevent dialogue policies from repeatedly making the same mistake, DDR also performs dialogue data augmentation by adding relevant experiences containing dead-end states. We first validate the dead-end detection reliability and then demonstrate the effectiveness and generality of the method by reporting experimental results on several dialogue datasets from different domains.