# Integrating Pretrained Language Model for Dialogue Policy Learning

arXiv: https://arxiv.org/abs/2111.01398 · HTML: https://mlsanity.com/paper/2111.01398/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 19 references that are in the map.
- First posted on arXiv 2 Nov 2021.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [Neural Approaches to Conversational AI](https://mlsanity.com/paper/1809.08267/) (2018)
- [Few-shot Natural Language Generation for Task-Oriented Dialog](https://mlsanity.com/paper/2002.12328/) (2020)
- [Unsupervised Text Style Transfer using Language Models as Discriminators](https://mlsanity.com/paper/1805.11749/) (2018)
- [ConvLab-2: An Open-Source Toolkit for Building, Evaluating, and Diagnosing Dialogue Systems](https://mlsanity.com/paper/2002.04793/) (2020)
- [Composite Task-Completion Dialogue Policy Learning via Hierarchical Deep Reinforcement Learning](https://mlsanity.com/paper/1704.03084/) (2017)
- [Guided Dialog Policy Learning: Reward Estimation for Multi-Domain Task-Oriented Dialog](https://mlsanity.com/paper/1908.10719/) (2019)
- [Deep Dyna-Q: Integrating Planning for Task-Completion Dialogue Policy Learning](https://mlsanity.com/paper/1801.06176/) (2018)
- [Switch-based Active Deep Dyna-Q: Efficient Adaptive Planning for Task-Completion Dialogue Policy Learning](https://mlsanity.com/paper/1811.07550/) (2018)
- [Adversarial Advantage Actor-Critic Model for Task-Completion Dialogue Policy Learning](https://mlsanity.com/paper/1710.11277/) (2017)
- [Learning Dialog Policies from Weak Demonstrations](https://mlsanity.com/paper/2004.11054/) (2020)
- [Guided Dialog Policy Learning without Adversarial Learning in the Loop](https://mlsanity.com/paper/2004.03267/) (2020)

## Built on by (most cited first, 5 of 5)

- [Acting Less is Reasoning More! Teaching Model to Act Efficiently](https://mlsanity.com/paper/2504.14870/) (2025)
- [Improving Factual Consistency for Knowledge-Grounded Dialogue Systems via Knowledge Enhancement and Alignment](https://mlsanity.com/paper/2310.08372/) (2023)
- [Rethinking Stateful Tool Use in Multi-Turn Dialogues: Benchmarks and Challenges](https://mlsanity.com/paper/2505.13328/) (2025)
- [SeRTS: Self-Rewarding Tree Search for Biomedical Retrieval-Augmented Generation](https://mlsanity.com/paper/2406.11258/) (2024)
- [JoTR: A Joint Transformer and Reinforcement Learning Framework for Dialog Policy Learning](https://mlsanity.com/paper/2309.00230/) (2023)

## Abstract (arXiv)

Reinforcement Learning (RL) has been witnessed its potential for training a dialogue policy agent towards maximizing the accumulated rewards given from users. However, the reward can be very sparse for it is usually only provided at the end of a dialog session, which causes unaffordable interaction requirements for an acceptable dialog agent. Distinguished from many efforts dedicated to optimizing the policy and recovering the reward alternatively which suffers from easily getting stuck in local optima and model collapse, we decompose the adversarial training into two steps: 1) we integrate a pre-trained language model as a discriminator to judge whether the current system action is good enough for the last user action (i.e., textitnext action prediction); 2) the discriminator gives and extra local dense reward to guide the agent's exploration. The experimental result demonstrates that our method significantly improves the complete rate (~4.4%) and success rate (~8.0%) of the dialogue system.