# Guided Dialog Policy Learning: Reward Estimation for Multi-Domain Task-Oriented Dialog

arXiv: https://arxiv.org/abs/1908.10719 · HTML: https://mlsanity.com/paper/1908.10719/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Aug 2019.

## Built on by (most cited first, 30 of 37)

- [Aligning Large Language Models with Human Preferences through Representation Engineering](https://mlsanity.com/paper/2312.15997/) (2023)
- [Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions](https://mlsanity.com/paper/2109.05794/) (2021)
- [A Dense Reward View on Aligning Text-to-Image Diffusion with Preference](https://mlsanity.com/paper/2402.08265/) (2024)
- [Preference-grounded Token-level Guidance for Language Model Fine-tuning](https://mlsanity.com/paper/2306.00398/) (2023)
- [A Survey on Recent Advances and Challenges in Reinforcement Learning Methods for Task-Oriented Dialogue Policy Learning](https://mlsanity.com/paper/2202.13675/) (2022)
- [Domain-independent User Simulation with Transformers for Task-oriented Dialogue Systems](https://mlsanity.com/paper/2106.08838/) (2021)
- [Fantastic Rewards and How to Tame Them: A Case Study on Reward Learning for Task-oriented Dialogue Systems](https://mlsanity.com/paper/2302.10342/) (2023)
- [A Survey on Deep Reinforcement Learning for Audio-Based Applications](https://mlsanity.com/paper/2101.00240/) (2021)
- [Offline Reinforcement Learning from Human Feedback in Real-World Sequence-to-Sequence Tasks](https://mlsanity.com/paper/2011.02511/) (2020)
- [Dynamic Dialogue Policy for Continual Reinforcement Learning](https://mlsanity.com/paper/2204.05928/) (2022)
- [Segmenting Text and Learning Their Rewards for Improved RLHF in Language Model](https://mlsanity.com/paper/2501.02790/) (2025)
- [Rethinking Stateful Tool Use in Multi-Turn Dialogues: Benchmarks and Challenges](https://mlsanity.com/paper/2505.13328/) (2025)
- [Mars: Modeling Context & State Representations with Contrastive Learning for End-to-End Task-Oriented Dialog](https://mlsanity.com/paper/2210.08917/) (2022)
- [Quantifying the Utility of User Simulators for Building Collaborative LLM Assistants](https://mlsanity.com/paper/2605.09808/) (2026)
- [Uncertainty Measures in Neural Belief Tracking and the Effects on Dialogue Policy Performance](https://mlsanity.com/paper/2109.04349/) (2021)
- [DEUS: A Data-driven Approach to Estimate User Satisfaction in Multi-turn Dialogues](https://mlsanity.com/paper/2103.01287/) (2021)
- [Imperfect also Deserves Reward: Multi-Level and Sequential Reward Modeling for Better Dialog Management](https://mlsanity.com/paper/2104.04748/) (2021)
- [Enhancing End-to-End Multi-Task Dialogue Systems: A Study on Intrinsic Motivation Reinforcement Learning Algorithms for Improved Training and Adaptability](https://mlsanity.com/paper/2401.18040/) (2024)
- [Integrating Pretrained Language Model for Dialogue Policy Learning](https://mlsanity.com/paper/2111.01398/) (2021)
- [Post-processing Networks: Method for Optimizing Pipeline Task-oriented Dialogue Systems using Reinforcement Learning](https://mlsanity.com/paper/2207.12185/) (2022)
- [High-Quality Diversification for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2106.00891/) (2021)
- ["Think Before You Speak": Improving Multi-Action Dialog Policy by Planning Single-Action Dialogs](https://mlsanity.com/paper/2204.11481/) (2022)
- [An Exploratory Study of AI System Risk Assessment from the Lens of Data Distribution and Uncertainty](https://mlsanity.com/paper/2212.06828/) (2022)
- [Rewarding What Matters: Step-by-Step Reinforcement Learning for Task-Oriented Dialogue](https://mlsanity.com/paper/2406.14457/) (2024)
- [WeaSuL: Weakly Supervised Dialogue Policy Learning: Reward Estimation for Multi-turn Dialogue](https://mlsanity.com/paper/2108.01487/) (2021)
- [Prior Lessons of Incremental Dialogue and Robot Action Management for the Age of Language Models](https://mlsanity.com/paper/2501.00953/) (2025)
- [SalesRLAgent: A Reinforcement Learning Approach for Real-Time Sales Conversion Prediction and Optimization](https://mlsanity.com/paper/2503.23303/) (2025)
- [Joint System-Wise Optimization for Pipeline Goal-Oriented Dialog System](https://mlsanity.com/paper/2106.04835/) (2021)
- [A Review of Dialogue Systems: From Trained Monkeys to Stochastic Parrots](https://mlsanity.com/paper/2111.01414/) (2021)
- [Workflow-Guided Response Generation for Task-Oriented Dialogue](https://mlsanity.com/paper/2311.08300/) (2023)

## Abstract (arXiv)

Dialog policy decides what and how a task-oriented dialog system will respond, and plays a vital role in delivering effective conversations. Many studies apply Reinforcement Learning to learn a dialog policy with the reward function which requires elaborate design and pre-specified user goals. With the growing needs to handle complex goals across multiple domains, such manually designed reward functions are not affordable to deal with the complexity of real-world tasks. To this end, we propose Guided Dialog Policy Learning, a novel algorithm based on Adversarial Inverse Reinforcement Learning for joint reward estimation and policy optimization in multi-domain task-oriented dialog. The proposed approach estimates the reward signal and infers the user goal in the dialog sessions. The reward estimator evaluates the state-action pairs so that it can guide the dialog policy at each dialog turn. Extensive experiments on a multi-domain dialog dataset show that the dialog policy guided by the learned reward function achieves remarkably higher task success than state-of-the-art baselines.