# Sample-efficient Actor-Critic Reinforcement Learning with Supervised Data for Dialogue Management

arXiv: https://arxiv.org/abs/1707.00130 · HTML: https://mlsanity.com/paper/1707.00130/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jul 2017.

## Built on by (most cited first, 18 of 18)

- [Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog](https://mlsanity.com/paper/1907.00456/) (2019)
- [GALAXY: A Generative Pre-trained Model for Task-Oriented Dialog with Semi-Supervised Learning and Explicit Policy Injection](https://mlsanity.com/paper/2111.14592/) (2021)
- [Human-centric Dialog Training via Offline Reinforcement Learning](https://mlsanity.com/paper/2010.05848/) (2020)
- [Transferable Dialogue Systems and User Simulators](https://mlsanity.com/paper/2107.11904/) (2021)
- [CHAI: A CHatbot AI for Task-Oriented Dialogue with Offline Reinforcement Learning](https://mlsanity.com/paper/2204.08426/) (2022)
- [A Survey on Recent Advances and Challenges in Reinforcement Learning Methods for Task-Oriented Dialogue Policy Learning](https://mlsanity.com/paper/2202.13675/) (2022)
- [Context-Aware Language Modeling for Goal-Oriented Dialogue Systems](https://mlsanity.com/paper/2204.10198/) (2022)
- [A Survey on Deep Reinforcement Learning for Audio-Based Applications](https://mlsanity.com/paper/2101.00240/) (2021)
- [Data Augmentation with Paraphrase Generation and Entity Extraction for Multimodal Dialogue System](https://mlsanity.com/paper/2205.04006/) (2022)
- [Perceiving the World: Question-guided Reinforcement Learning for Text-based Games](https://mlsanity.com/paper/2204.09597/) (2022)
- [NLU for Game-based Learning in Real: Initial Evaluations](https://mlsanity.com/paper/2205.13754/) (2022)
- [End-to-End Evaluation of a Spoken Dialogue System for Learning Basic Mathematics](https://mlsanity.com/paper/2211.03511/) (2022)
- [Successor Features for Efficient Multisubject Controlled Text Generation](https://mlsanity.com/paper/2311.04921/) (2023)
- [An empirical assessment of deep learning approaches to task-oriented dialog management](https://mlsanity.com/paper/2108.03478/) (2021)
- [Taming Continuous Posteriors for Latent Variational Dialogue Policies](https://mlsanity.com/paper/2205.07633/) (2022)
- [Reinforcement Explanation Learning](https://mlsanity.com/paper/2111.13406/) (2021)
- [A comparison of RL-based and PID controllers for 6-DOF swimming robots: hybrid underwater object tracking](https://mlsanity.com/paper/2401.16618/) (2024)
- [Architectural Transformations and Emerging Verification Demands in AI-Enabled Cyber-Physical Systems](https://mlsanity.com/paper/2510.00519/) (2025)

## Abstract (arXiv)

Deep reinforcement learning (RL) methods have significant potential for dialogue policy optimisation. However, they suffer from a poor performance in the early stages of learning. This is especially problematic for on-line learning with real users. Two approaches are introduced to tackle this problem. Firstly, to speed up the learning process, two sample-efficient neural networks algorithms: trust region actor-critic with experience replay (TRACER) and episodic natural actor-critic with experience replay (eNACER) are presented. For TRACER, the trust region helps to control the learning step size and avoid catastrophic model changes. For eNACER, the natural gradient identifies the steepest ascent direction in policy space to speed up the convergence. Both models employ off-policy learning with experience replay to improve sample-efficiency. Secondly, to mitigate the cold start issue, a corpus of demonstration data is utilised to pre-train the models prior to on-line reinforcement learning. Combining these two approaches, we demonstrate a practical approach to learn deep RL-based dialogue policies and demonstrate their effectiveness in a task-oriented information seeking domain.