# Modelling Hierarchical Structure between Dialogue Policy and Natural Language Generator with Option Framework for Task-oriented Dialogue System

arXiv: https://arxiv.org/abs/2006.06814 · HTML: https://mlsanity.com/paper/2006.06814/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jun 2020.

## Built on by (most cited first, 28 of 28)

- [Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization](https://mlsanity.com/paper/2406.01171/) (2024)
- [GALAXY: A Generative Pre-trained Model for Task-Oriented Dialog with Semi-Supervised Learning and Explicit Policy Injection](https://mlsanity.com/paper/2111.14592/) (2021)
- [Shades of BLEU, Flavours of Success: The Case of MultiWOZ](https://mlsanity.com/paper/2106.05555/) (2021)
- [CHAI: A CHatbot AI for Task-Oriented Dialogue with Offline Reinforcement Learning](https://mlsanity.com/paper/2204.08426/) (2022)
- [Unlocking the Potential of User Feedback: Leveraging Large Language Model as User Simulator to Enhance Dialogue System](https://mlsanity.com/paper/2306.09821/) (2023)
- [Annotation Inconsistency and Entity Bias in MultiWOZ](https://mlsanity.com/paper/2105.14150/) (2021)
- [A Survey on Recent Advances and Challenges in Reinforcement Learning Methods for Task-Oriented Dialogue Policy Learning](https://mlsanity.com/paper/2202.13675/) (2022)
- [Task-Optimized Adapters for an End-to-End Task-Oriented Dialogue System](https://mlsanity.com/paper/2305.02468/) (2023)
- [Context-Aware Language Modeling for Goal-Oriented Dialogue Systems](https://mlsanity.com/paper/2204.10198/) (2022)
- [BORT: Back and Denoising Reconstruction for End-to-End Task-Oriented Dialog](https://mlsanity.com/paper/2205.02471/) (2022)
- [Interpretable Math Word Problem Solution Generation Via Step-by-step Planning](https://mlsanity.com/paper/2306.00784/) (2023)
- [End-to-end Task-oriented Dialogue: A Survey of Tasks, Methods, and Future Directions](https://mlsanity.com/paper/2311.09008/) (2023)
- [Simulation-Free Hierarchical Latent Policy Planning for Proactive Dialogues](https://mlsanity.com/paper/2412.14584/) (2024)
- [Database Search Results Disambiguation for Task-Oriented Dialog Systems](https://mlsanity.com/paper/2112.08351/) (2021)
- [Retrieve & Memorize: Dialog Policy Learning with Multi-Action Memory](https://mlsanity.com/paper/2106.02317/) (2021)
- [AARGH! End-to-end Retrieval-Generation for Task-Oriented Dialog](https://mlsanity.com/paper/2209.03632/) (2022)
- [Enhancing Large Language Model Induced Task-Oriented Dialogue Systems Through Look-Forward Motivated Goals](https://mlsanity.com/paper/2309.08949/) (2023)
- [DiactTOD: Learning Generalizable Latent Dialogue Acts for Controllable Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2308.00878/) (2023)
- [DFA-RAG: Conversational Semantic Router for Large Language Model with Definite Finite Automaton](https://mlsanity.com/paper/2402.04411/) (2024)
- [Turning Conversations into Workflows: A Framework to Extract and Evaluate Dialog Workflows for Service AI Agents](https://mlsanity.com/paper/2502.17321/) (2025)
- [Causal-aware Safe Policy Improvement for Task-oriented dialogue](https://mlsanity.com/paper/2103.06370/) (2021)
- [TOD-Flow: Modeling the Structure of Task-Oriented Dialogues](https://mlsanity.com/paper/2312.04668/) (2023)
- [DORA: Toward Policy Optimization for Task-oriented Dialogue System with Efficient Context](https://mlsanity.com/paper/2107.03286/) (2021)
- [UBARv2: Towards Mitigating Exposure Bias in Task-Oriented Dialogs](https://mlsanity.com/paper/2209.07239/) (2022)
- [Replicating Complex Dialogue Policy of Humans via Offline Imitation Learning with Supervised Regularization](https://mlsanity.com/paper/2305.03987/) (2023)
- [A Unified Framework for Slot based Response Generation in a Multimodal Dialogue System](https://mlsanity.com/paper/2305.17433/) (2023)
- [Collaborative and Proactive Management of Task-Oriented Conversations](https://mlsanity.com/paper/2510.05110/) (2025)
- [Impact of enriched meaning representations for language generation in dialogue tasks: A comprehensive exploration of the relevance of tasks, corpora and metrics](https://mlsanity.com/paper/2603.29518/) (2026)

## Abstract (arXiv)

Designing task-oriented dialogue systems is a challenging research topic, since it needs not only to generate utterances fulfilling user requests but also to guarantee the comprehensibility. Many previous works trained end-to-end (E2E) models with supervised learning (SL), however, the bias in annotated system utterances remains as a bottleneck. Reinforcement learning (RL) deals with the problem through using non-differentiable evaluation metrics (e.g., the success rate) as rewards. Nonetheless, existing works with RL showed that the comprehensibility of generated system utterances could be corrupted when improving the performance on fulfilling user requests. In our work, we (1) propose modelling the hierarchical structure between dialogue policy and natural language generator (NLG) with the option framework, called HDNO, where the latent dialogue act is applied to avoid designing specific dialogue act representations; (2) train HDNO via hierarchical reinforcement learning (HRL), as well as suggest the asynchronous updates between dialogue policy and NLG during training to theoretically guarantee their convergence to a local maximizer; and (3) propose using a discriminator modelled with language models as an additional reward to further improve the comprehensibility. We test HDNO on MultiWoz 2.0 and MultiWoz 2.1, the datasets on multi-domain dialogues, in comparison with word-level E2E model trained with RL, LaRL and HDSA, showing improvements on the performance evaluated by automatic evaluation metrics and human evaluation. Finally, we demonstrate the semantic meanings of latent dialogue acts to show the explanability for HDNO.