# Multi-Agent Task-Oriented Dialog Policy Learning with Role-Aware Reward Decomposition

arXiv: https://arxiv.org/abs/2004.03809 · HTML: https://mlsanity.com/paper/2004.03809/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Apr 2020.

## Built on by (most cited first, 23 of 23)

- [A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems](https://mlsanity.com/paper/2402.18013/) (2024)
- [Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey](https://mlsanity.com/paper/2105.04387/) (2021)
- [Know You First and Be You Better: Modeling Human-Like User Simulators via Implicit Profiles](https://mlsanity.com/paper/2502.18968/) (2025)
- [Transferable Dialogue Systems and User Simulators](https://mlsanity.com/paper/2107.11904/) (2021)
- [Planning Like Human: A Dual-process Framework for Dialogue Planning](https://mlsanity.com/paper/2406.05374/) (2024)
- [A Survey on Recent Advances and Challenges in Reinforcement Learning Methods for Task-Oriented Dialogue Policy Learning](https://mlsanity.com/paper/2202.13675/) (2022)
- [PlatoLM: Teaching LLMs in Multi-Round Dialogue via a User Simulator](https://mlsanity.com/paper/2308.11534/) (2023)
- [Don't be Contradicted with Anything! CI-ToD: Towards Benchmarking Consistency for Task-oriented Dialogue System](https://mlsanity.com/paper/2109.11292/) (2021)
- [Is MultiWOZ a Solved Task? An Interactive TOD Evaluation Framework with User Simulator](https://mlsanity.com/paper/2210.14529/) (2022)
- [KRLS: Improving End-to-End Response Generation in Task Oriented Dialog with Reinforced Keywords Learning](https://mlsanity.com/paper/2211.16773/) (2022)
- [A Generative User Simulator with GPT-based Architecture and Goal State Tracking for Reinforced Multi-Domain Dialog Systems](https://mlsanity.com/paper/2210.08692/) (2022)
- [DIAL: Direct Iterative Adversarial Learning for Realistic Multi-Turn Dialogue Simulation](https://mlsanity.com/paper/2512.20773/) (2025)
- [High-Quality Diversification for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2106.00891/) (2021)
- [Rewarding What Matters: Step-by-Step Reinforcement Learning for Task-Oriented Dialogue](https://mlsanity.com/paper/2406.14457/) (2024)
- [WeaSuL: Weakly Supervised Dialogue Policy Learning: Reward Estimation for Multi-turn Dialogue](https://mlsanity.com/paper/2108.01487/) (2021)
- [Reinforced Approximate Exploratory Data Analysis](https://mlsanity.com/paper/2212.06225/) (2022)
- [Rescue Conversations from Dead-ends: Efficient Exploration for Task-oriented Dialogue Policy Optimization](https://mlsanity.com/paper/2305.03262/) (2023)
- [A Task-oriented Dialog Model with Task-progressive and Policy-aware Pre-training](https://mlsanity.com/paper/2310.00597/) (2023)
- [Tailored Conversations beyond LLMs: A RL-Based Dialogue Manager](https://mlsanity.com/paper/2506.19652/) (2025)
- [Jointly Reinforced User Simulator and Task-oriented Dialog System with Simplified Generative Architecture](https://mlsanity.com/paper/2210.06706/) (2022)
- [An Asynchronous Updating Reinforcement Learning Framework for Task-oriented Dialog System](https://mlsanity.com/paper/2305.02718/) (2023)
- [Why Guided Dialog Policy Learning performs well? Understanding the role of adversarial learning and its alternative](https://mlsanity.com/paper/2307.06721/) (2023)
- [JoTR: A Joint Transformer and Reinforcement Learning Framework for Dialog Policy Learning](https://mlsanity.com/paper/2309.00230/) (2023)

## Abstract (arXiv)

Many studies have applied reinforcement learning to train a dialog policy and show great promise these years. One common approach is to employ a user simulator to obtain a large number of simulated user experiences for reinforcement learning algorithms. However, modeling a realistic user simulator is challenging. A rule-based simulator requires heavy domain expertise for complex tasks, and a data-driven simulator requires considerable data and it is even unclear how to evaluate a simulator. To avoid explicitly building a user simulator beforehand, we propose Multi-Agent Dialog Policy Learning, which regards both the system and the user as the dialog agents. Two agents interact with each other and are jointly learned simultaneously. The method uses the actor-critic framework to facilitate pretraining and improve scalability. We also propose Hybrid Value Network for the role-aware reward decomposition to integrate role-specific domain knowledge of each agent in the task-oriented dialog. Results show that our method can successfully build a system policy and a user policy simultaneously, and two agents can achieve a high task success rate through conversational interaction.