# "Think Before You Speak": Improving Multi-Action Dialog Policy by Planning Single-Action Dialogs

arXiv: https://arxiv.org/abs/2204.11481 · HTML: https://mlsanity.com/paper/2204.11481/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 26 references that are in the map.
- First posted on arXiv 25 Apr 2022.

## Builds on (most cited first, 19 of 19)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset](https://mlsanity.com/paper/1909.05855/) (2019)
- [Sample Efficient Actor-Critic with Experience Replay](https://mlsanity.com/paper/1611.01224/) (2016)
- [Recent Advances and Challenges in Task-oriented Dialog System](https://mlsanity.com/paper/2003.07490/) (2020)
- [End-to-End Task-Completion Neural Dialogue Systems](https://mlsanity.com/paper/1703.01008/) (2017)
- [ConvLab-2: An Open-Source Toolkit for Building, Evaluating, and Diagnosing Dialogue Systems](https://mlsanity.com/paper/2002.04793/) (2020)
- [Guided Dialog Policy Learning: Reward Estimation for Multi-Domain Task-Oriented Dialog](https://mlsanity.com/paper/1908.10719/) (2019)
- [Deep Dyna-Q: Integrating Planning for Task-Completion Dialogue Policy Learning](https://mlsanity.com/paper/1801.06176/) (2018)
- [Adversarial Learning of Task-Oriented Neural Dialog Models](https://mlsanity.com/paper/1805.11762/) (2018)
- [Switch-based Active Deep Dyna-Q: Efficient Adaptive Planning for Task-Completion Dialogue Policy Learning](https://mlsanity.com/paper/1811.07550/) (2018)
- [Discriminative Deep Dyna-Q: Robust Planning for Dialogue Policy Learning](https://mlsanity.com/paper/1808.09442/) (2018)
- [Learning Dialog Policies from Weak Demonstrations](https://mlsanity.com/paper/2004.11054/) (2020)
- [Rethinking Supervised Learning and Reinforcement Learning in Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2009.09781/) (2020)
- [Semi-Supervised Dialogue Policy Learning via Stochastic Reward Estimation](https://mlsanity.com/paper/2005.04379/) (2020)
- [Modeling Multi-Action Policy for Task-Oriented Dialogues](https://mlsanity.com/paper/1908.11546/) (2019)
- [Learning to Check Contract Inconsistencies](https://mlsanity.com/paper/2012.08150/) (2020)
- [Stable deep reinforcement learning method by predicting uncertainty in rewards as a subtask](https://mlsanity.com/paper/2101.06906/) (2021)

## Built on by (most cited first, 3 of 3)

- [Prompt-Based Monte-Carlo Tree Search for Goal-Oriented Dialogue Policy Planning](https://mlsanity.com/paper/2305.13660/) (2023)
- [Multi-Action Dialog Policy Learning from Logged User Feedback](https://mlsanity.com/paper/2302.13505/) (2023)
- [Improving Dialogue Management: Quality Datasets vs Models](https://mlsanity.com/paper/2310.01339/) (2023)

## Abstract (arXiv)

Multi-action dialog policy (MADP), which generates multiple atomic dialog actions per turn, has been widely applied in task-oriented dialog systems to provide expressive and efficient system responses. Existing MADP models usually imitate action combinations from the labeled multi-action dialog samples. Due to data limitations, they generalize poorly toward unseen dialog flows. While interactive learning and reinforcement learning algorithms can be applied to incorporate external data sources of real users and user simulators, they take significant manual effort to build and suffer from instability. To address these issues, we propose Planning Enhanced Dialog Policy (PEDP), a novel multi-task learning framework that learns single-action dialog dynamics to enhance multi-action prediction. Our PEDP method employs model-based planning for conceiving what to express before deciding the current response through simulating single-action dialogs. Experimental results on the MultiWOZ dataset demonstrate that our fully supervised learning-based method achieves a solid task success rate of 90.6%, improving 3% compared to the state-of-the-art methods.