# Training an adaptive dialogue policy for interactive learning of visually grounded word meanings

arXiv: https://arxiv.org/abs/1709.10426 · HTML: https://mlsanity.com/paper/1709.10426/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Sep 2017.

## Built on by (most cited first, 1 of 1)

- [Learning to generate and corr- uh I mean repair language in real-time](https://mlsanity.com/paper/2308.11683/) (2023)

## Abstract (arXiv)

We present a multi-modal dialogue system for interactive learning of perceptually grounded word meanings from a human tutor. The system integrates an incremental, semantic parsing/generation framework - Dynamic Syntax and Type Theory with Records (DS-TTR) - with a set of visual classifiers that are learned throughout the interaction and which ground the meaning representations that it produces. We use this system in interaction with a simulated human tutor to study the effects of different dialogue policies and capabilities on the accuracy of learned meanings, learning rates, and efforts/costs to the tutor. We show that the overall performance of the learning agent is affected by (1) who takes initiative in the dialogues; (2) the ability to express/use their confidence level about visual attributes; and (3) the ability to process elliptical and incrementally constructed dialogue turns. Ultimately, we train an adaptive dialogue policy which optimises the trade-off between classifier accuracy and tutoring costs.