# Evaluate On-the-job Learning Dialogue Systems and a Case Study for Natural Language Understanding

arXiv: https://arxiv.org/abs/2102.13589 · HTML: https://mlsanity.com/paper/2102.13589/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 16 references that are in the map.
- First posted on arXiv 26 Feb 2021.

## Builds on (most cited first, 6 of 6)

- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Survey on Evaluation Methods for Dialogue Systems](https://mlsanity.com/paper/1905.04071/) (2019)
- [Learning from Dialogue after Deployment: Feed Yourself, Chatbot!](https://mlsanity.com/paper/1901.05415/) (2019)
- [Lifelong and Interactive Learning of Factual Knowledge in Dialogues](https://mlsanity.com/paper/1907.13295/) (2019)
- [Building an Application Independent Natural Language Interface](https://mlsanity.com/paper/1910.14084/) (2019)

## Built on by (most cited first, 2 of 2)

- [Learning From Free-Text Human Feedback -- Collect New Datasets Or Extend Existing Ones?](https://mlsanity.com/paper/2310.15758/) (2023)
- [Learning from Implicit User Feedback, Emotions and Demographic Information in Task-Oriented and Document-Grounded Dialogues](https://mlsanity.com/paper/2401.09248/) (2024)

## Abstract (arXiv)

On-the-job learning consists in continuously learning while being used in production, in an open environment, meaning that the system has to deal on its own with situations and elements never seen before. The kind of systems that seem to be especially adapted to on-the-job learning are dialogue systems, since they can take advantage of their interactions with users to collect feedback to adapt and improve their components over time. Some dialogue systems performing on-the-job learning have been built and evaluated but no general methodology has yet been defined. Thus in this paper, we propose a first general methodology for evaluating on-the-job learning dialogue systems. We also describe a task-oriented dialogue system which improves on-the-job its natural language component through its user interactions. We finally evaluate our system with the described methodology.