# On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systems

arXiv: https://arxiv.org/abs/1605.07669 · HTML: https://mlsanity.com/paper/1605.07669/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 May 2016.

## Built on by (most cited first, 28 of 28)

- [Offline RL for Natural Language Generation with Implicit Language Q Learning](https://mlsanity.com/paper/2206.11871/) (2022)
- [DialFRED: Dialogue-Enabled Agents for Embodied Instruction Following](https://mlsanity.com/paper/2202.13330/) (2022)
- [Neural Approaches to Conversational Information Retrieval](https://mlsanity.com/paper/2201.05176/) (2022)
- [Gradient Imitation Reinforcement Learning for Low Resource Relation Extraction](https://mlsanity.com/paper/2109.06415/) (2021)
- [A Survey on Recent Advances and Challenges in Reinforcement Learning Methods for Task-Oriented Dialogue Policy Learning](https://mlsanity.com/paper/2202.13675/) (2022)
- [Deep Dyna-Q: Integrating Planning for Task-Completion Dialogue Policy Learning](https://mlsanity.com/paper/1801.06176/) (2018)
- [A Survey on Deep Reinforcement Learning for Audio-Based Applications](https://mlsanity.com/paper/2101.00240/) (2021)
- [Multi-User Chat Assistant (MUCA): a Framework Using LLMs to Facilitate Group Conversations](https://mlsanity.com/paper/2401.04883/) (2024)
- [Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challenges](https://mlsanity.com/paper/2203.10012/) (2022)
- [Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges](https://mlsanity.com/paper/2308.00031/) (2023)
- [Toward Self-learning End-to-End Task-Oriented Dialog Systems](https://mlsanity.com/paper/2201.06849/) (2022)
- [Optimizing Conversational Quality in Spoken Dialogue Systems with Reinforcement Learning from AI Feedback](https://mlsanity.com/paper/2601.19063/) (2026)
- [Task-oriented Dialogue Systems: performance vs. quality-optima, a review](https://mlsanity.com/paper/2112.11176/) (2021)
- [Discovering Generalizable Spatial Goal Representations via Graph-based Active Reward Learning](https://mlsanity.com/paper/2211.15339/) (2022)
- [How to Evaluate Your Dialogue Models: A Review of Approaches](https://mlsanity.com/paper/2108.01369/) (2021)
- [Prior Lessons of Incremental Dialogue and Robot Action Management for the Age of Language Models](https://mlsanity.com/paper/2501.00953/) (2025)
- [Deep Reinforcement Learning for Optimal Portfolio Allocation: A Comparative Study with Mean-Variance Optimization](https://mlsanity.com/paper/2602.17098/) (2026)
- [Rescue Conversations from Dead-ends: Efficient Exploration for Task-oriented Dialogue Policy Optimization](https://mlsanity.com/paper/2305.03262/) (2023)
- [Conversational Assistants to support Heart Failure Patients: comparing a Neurosymbolic Architecture with ChatGPT](https://mlsanity.com/paper/2504.17753/) (2025)
- [One Model, Multiple Goals: Adaptive Multi-Objective Learning for E-commerce Dialogue Systems](https://mlsanity.com/paper/2606.09293/) (2026)
- [Findings from Experiments of On-line Joint Reinforcement Learning of Semantic Parser and Dialogue Manager with real Users](https://mlsanity.com/paper/2110.13213/) (2021)
- [Why Guided Dialog Policy Learning performs well? Understanding the role of adversarial learning and its alternative](https://mlsanity.com/paper/2307.06721/) (2023)
- [JoTR: A Joint Transformer and Reinforcement Learning Framework for Dialog Policy Learning](https://mlsanity.com/paper/2309.00230/) (2023)
- [Evaluating Task-Oriented Dialogue Consistency through Constraint Satisfaction](https://mlsanity.com/paper/2407.11857/) (2024)
- [A Stack-Propagation Framework for Low-Resource Personalized Dialogue Generation](https://mlsanity.com/paper/2410.20174/) (2024)
- [Personality-Aware Reinforcement Learning for Persuasive Dialogue with LLM-Driven Simulation](https://mlsanity.com/paper/2601.06877/) (2026)
- [Towards Detecting Inconsistencies in End-to-end Generated TODs](https://mlsanity.com/paper/2607.09338/) (2026)
- [Not All Speech Is Intent: Adaptive Self-Correcting Inference Layer for Post-ASR False Wake-Up](https://mlsanity.com/paper/2609.12469/) (2026)

## Abstract (arXiv)

The ability to compute an accurate reward function is essential for optimising a dialogue policy via reinforcement learning. In real-world applications, using explicit user feedback as the reward signal is often unreliable and costly to collect. This problem can be mitigated if the user's intent is known in advance or data is available to pre-train a task success predictor off-line. In practice neither of these apply for most real world applications. Here we propose an on-line learning framework whereby the dialogue policy is jointly trained alongside the reward model via active learning with a Gaussian process model. This Gaussian process operates on a continuous space dialogue representation generated in an unsupervised fashion using a recurrent neural network encoder-decoder. The experimental results demonstrate that the proposed framework is able to significantly reduce data annotation costs and mitigate noisy user feedback in dialogue policy learning.