# Findings from Experiments of On-line Joint Reinforcement Learning of Semantic Parser and Dialogue Manager with real Users

arXiv: https://arxiv.org/abs/2110.13213 · HTML: https://mlsanity.com/paper/2110.13213/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 58 references that are in the map.
- First posted on arXiv 25 Oct 2021.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [A Network-based End-to-End Trainable Task-oriented Dialogue System](https://mlsanity.com/paper/1604.04562/) (2016)
- [TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents](https://mlsanity.com/paper/1901.08149/) (2019)
- [FlauBERT: Unsupervised Language Model Pre-training for French](https://mlsanity.com/paper/1912.05372/) (2019)
- [Learning from Dialogue after Deployment: Feed Yourself, Chatbot!](https://mlsanity.com/paper/1901.05415/) (2019)
- [Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models](https://mlsanity.com/paper/1902.08858/) (2019)
- [End-to-End Task-Completion Neural Dialogue Systems](https://mlsanity.com/paper/1703.01008/) (2017)
- [Dialogue Learning With Human-In-The-Loop](https://mlsanity.com/paper/1611.09823/) (2016)
- [Towards Zero-Shot Frame Semantic Parsing for Domain Scaling](https://mlsanity.com/paper/1707.02363/) (2017)
- [Towards End-to-End Reinforcement Learning of Dialogue Agents for Information Access](https://mlsanity.com/paper/1609.00777/) (2016)
- [On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systems](https://mlsanity.com/paper/1605.07669/) (2016)
- [Zero-Shot Dialog Generation with Cross-Domain Latent Actions](https://mlsanity.com/paper/1805.04803/) (2018)
- [Latent Intention Dialogue Models](https://mlsanity.com/paper/1705.10229/) (2017)
- [Multi-task learning for Joint Language Understanding and Dialogue State Tracking](https://mlsanity.com/paper/1811.05408/) (2018)
- [Kalman Temporal Differences](https://mlsanity.com/paper/1406.3270/) (2014)
- [A Benchmarking Environment for Reinforcement Learning Based Task Oriented Dialogue Management](https://mlsanity.com/paper/1711.11023/) (2017)
- [End-to-End Joint Learning of Natural Language Understanding and Dialogue Manager](https://mlsanity.com/paper/1612.00913/) (2016)
- [Jointly Learning Semantic Parser and Natural Language Generator via Dual Information Maximization](https://mlsanity.com/paper/1906.00575/) (2019)
- [Zero-Shot Learning for Semantic Utterance Classification](https://mlsanity.com/paper/1401.0509/) (2013)
- [Nearly Zero-Shot Learning for Semantic Decoding in Spoken Dialogue Systems](https://mlsanity.com/paper/1806.05484/) (2018)

## Abstract (arXiv)

Design of dialogue systems has witnessed many advances lately, yet acquiring huge set of data remains an hindrance to their fast development for a new task or language. Besides, training interactive systems with batch data is not satisfactory. On-line learning is pursued in this paper as a convenient way to alleviate these difficulties. After the system modules are initiated, a single process handles data collection, annotation and use in training algorithms. A new challenge is to control the cost of the on-line learning borne by the user. Our work focuses on learning the semantic parsing and dialogue management modules (speech recognition and synthesis offer ready-for-use solutions). In this context we investigate several variants of simultaneous learning which are tested in user trials. In our experiments, with varying merits, they can all achieve good performance with only a few hundreds of training dialogues and overstep a handcrafted system. The analysis of these experiments gives us some insights, discussed in the paper, into the difficulty for the system's trainers to establish a coherent and constant behavioural strategy to enable a fast and good-quality training phase.