# Reward Shaping with Recurrent Neural Networks for Speeding up On-Line Policy Learning in Spoken Dialogue Systems

arXiv: https://arxiv.org/abs/1508.03391 · HTML: https://mlsanity.com/paper/1508.03391/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Aug 2015.

## Built on by (most cited first, 9 of 9)

- [A Survey on Recent Advances and Challenges in Reinforcement Learning Methods for Task-Oriented Dialogue Policy Learning](https://mlsanity.com/paper/2202.13675/) (2022)
- [An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey](https://mlsanity.com/paper/2209.08890/) (2022)
- [Generative Conversational Networks](https://mlsanity.com/paper/2106.08484/) (2021)
- [Automated Reinforcement Learning: An Overview](https://mlsanity.com/paper/2201.05000/) (2022)
- [Imperfect also Deserves Reward: Multi-Level and Sequential Reward Modeling for Better Dialog Management](https://mlsanity.com/paper/2104.04748/) (2021)
- [Evaluating the progress of Deep Reinforcement Learning in the real world: aligning domain-agnostic and domain-specific research](https://mlsanity.com/paper/2107.03015/) (2021)
- [An empirical assessment of deep learning approaches to task-oriented dialog management](https://mlsanity.com/paper/2108.03478/) (2021)
- [JoTR: A Joint Transformer and Reinforcement Learning Framework for Dialog Policy Learning](https://mlsanity.com/paper/2309.00230/) (2023)
- [Pragmatist Intelligence: Where the Principle of Usefulness Can Take ANNs](https://mlsanity.com/paper/2405.04386/) (2024)

## Abstract (arXiv)

Statistical spoken dialogue systems have the attractive property of being able to be optimised from data via interactions with real users. However in the reinforcement learning paradigm the dialogue manager (agent) often requires significant time to explore the state-action space to learn to behave in a desirable manner. This is a critical issue when the system is trained on-line with real users where learning costs are expensive. Reward shaping is one promising technique for addressing these concerns. Here we examine three recurrent neural network (RNN) approaches for providing reward shaping information in addition to the primary (task-orientated) environmental feedback. These RNNs are trained on returns from dialogues generated by a simulated user and attempt to diffuse the overall evaluation of the dialogue back down to the turn level to guide the agent towards good behaviour faster. In both simulated and real user scenarios these RNNs are shown to increase policy learning speed. Importantly, they do not require prior knowledge of the user's goal.