# Feudal Reinforcement Learning for Dialogue Management in Large Domains

arXiv: https://arxiv.org/abs/1803.03232 · HTML: https://mlsanity.com/paper/1803.03232/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Mar 2018.

## Built on by (most cited first, 11 of 11)

- [A Survey on Recent Advances and Challenges in Reinforcement Learning Methods for Task-Oriented Dialogue Policy Learning](https://mlsanity.com/paper/2202.13675/) (2022)
- [A Survey on Deep Reinforcement Learning for Audio-Based Applications](https://mlsanity.com/paper/2101.00240/) (2021)
- [Dynamic Dialogue Policy for Continual Reinforcement Learning](https://mlsanity.com/paper/2204.05928/) (2022)
- [Imperfect also Deserves Reward: Multi-Level and Sequential Reward Modeling for Better Dialog Management](https://mlsanity.com/paper/2104.04748/) (2021)
- [Enhancing End-to-End Multi-Task Dialogue Systems: A Study on Intrinsic Motivation Reinforcement Learning Algorithms for Improved Training and Adaptability](https://mlsanity.com/paper/2401.18040/) (2024)
- [What Does The User Want? Information Gain for Hierarchical Dialogue Policy Optimisation](https://mlsanity.com/paper/2109.07129/) (2021)
- [Rewarding What Matters: Step-by-Step Reinforcement Learning for Task-Oriented Dialogue](https://mlsanity.com/paper/2406.14457/) (2024)
- [Graph Neural Network Policies and Imitation Learning for Multi-Domain Task-Oriented Dialogues](https://mlsanity.com/paper/2210.05252/) (2022)
- [Few-Shot Structured Policy Learning for Multi-Domain and Multi-Task Dialogues](https://mlsanity.com/paper/2302.11199/) (2023)
- [Feudal Reinforcement Learning by Reading Manuals](https://mlsanity.com/paper/2110.06477/) (2021)
- [Improving a sequence-to-sequence nlp model using a reinforcement learning policy algorithm](https://mlsanity.com/paper/2212.14117/) (2022)

## Abstract (arXiv)

Reinforcement learning (RL) is a promising approach to solve dialogue policy optimisation. Traditional RL algorithms, however, fail to scale to large domains due to the curse of dimensionality. We propose a novel Dialogue Management architecture, based on Feudal RL, which decomposes the decision into two steps; a first step where a master policy selects a subset of primitive actions, and a second step where a primitive action is chosen from the selected subset. The structural information included in the domain ontology is used to abstract the dialogue state space, taking the decisions at each step using different parts of the abstracted state. This, combined with an information sharing mechanism between slots, increases the scalability to large domains. We show that an implementation of this approach, based on Deep-Q Networks, significantly outperforms previous state of the art in several dialogue domains and environments, without the need of any additional reward signal.