# AgentGraph: Towards Universal Dialogue Management with Structured Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/1905.11259 · HTML: https://mlsanity.com/paper/1905.11259/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 May 2019.

## Built on by (most cited first, 12 of 12)

- [META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI](https://mlsanity.com/paper/2205.11029/) (2022)
- [Reinforced Neighborhood Selection Guided Multi-Relational Graph Neural Networks](https://mlsanity.com/paper/2104.07886/) (2021)
- [ShadowGNN: Graph Projection Neural Network for Text-to-SQL Parser](https://mlsanity.com/paper/2104.04689/) (2021)
- [End-to-end Task-oriented Dialogue: A Survey of Tasks, Methods, and Future Directions](https://mlsanity.com/paper/2311.09008/) (2023)
- [LET: Linguistic Knowledge Enhanced Graph Transformer for Chinese Short Text Matching](https://mlsanity.com/paper/2102.12671/) (2021)
- [Fast Inference for Augmented Large Language Models](https://mlsanity.com/paper/2410.18248/) (2024)
- [UniDU: Towards A Unified Generative Dialogue Understanding Framework](https://mlsanity.com/paper/2204.04637/) (2022)
- [Imperfect also Deserves Reward: Multi-Level and Sequential Reward Modeling for Better Dialog Management](https://mlsanity.com/paper/2104.04748/) (2021)
- [GraphWOZ: Dialogue Management with Conversational Knowledge Graphs](https://mlsanity.com/paper/2211.12852/) (2022)
- [TIE: Topological Information Enhanced Structural Reading Comprehension on Web Pages](https://mlsanity.com/paper/2205.06435/) (2022)
- [OPAL: Ontology-Aware Pretrained Language Model for End-to-End Task-Oriented Dialogue](https://mlsanity.com/paper/2209.04595/) (2022)
- [Dialogue Strategy Adaptation to New Action Sets Using Multi-dimensional Modelling](https://mlsanity.com/paper/2204.07082/) (2022)

## Abstract (arXiv)

Dialogue policy plays an important role in task-oriented spoken dialogue systems. It determines how to respond to users. The recently proposed deep reinforcement learning (DRL) approaches have been used for policy optimization. However, these deep models are still challenging for two reasons: 1) Many DRL-based policies are not sample-efficient. 2) Most models don't have the capability of policy transfer between different domains. In this paper, we propose a universal framework, AgentGraph, to tackle these two problems. The proposed AgentGraph is the combination of GNN-based architecture and DRL-based algorithm. It can be regarded as one of the multi-agent reinforcement learning approaches. Each agent corresponds to a node in a graph, which is defined according to the dialogue domain ontology. When making a decision, each agent can communicate with its neighbors on the graph. Under AgentGraph framework, we further propose Dual GNN-based dialogue policy, which implicitly decomposes the decision in each turn into a high-level global decision and a low-level local decision. Experiments show that AgentGraph models significantly outperform traditional reinforcement learning approaches on most of the 18 tasks of the PyDial benchmark. Moreover, when transferred from the source task to a target task, these models not only have acceptable initial performance but also converge much faster on the target task.