# Reference Resolution and Context Change in Multimodal Situated Dialogue for Exploring Data Visualizations

arXiv: https://arxiv.org/abs/2209.02215 · HTML: https://mlsanity.com/paper/2209.02215/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 47 references that are in the map.
- First posted on arXiv 6 Sep 2022.

## Builds on (most cited first, 7 of 7)

- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [Named Entity Recognition with Bidirectional LSTM-CNNs](https://mlsanity.com/paper/1511.08308/) (2015)
- [NL4DV: A Toolkit for Generating Analytic Specifications for Data Visualization from Natural Language Queries](https://mlsanity.com/paper/2008.10723/) (2020)
- [Transfer Learning for Sequence Tagging with Hierarchical Recurrent Networks](https://mlsanity.com/paper/1703.06345/) (2017)
- [FlowSense: A Natural Language Interface for Visual Data Exploration within a Dataflow System](https://mlsanity.com/paper/1908.00681/) (2019)
- [Structured prediction models for RNN based sequence labeling in clinical text](https://mlsanity.com/paper/1608.00612/) (2016)
- [Quda: Natural Language Queries for Visual Data Analytics](https://mlsanity.com/paper/2005.03257/) (2020)

## Abstract (arXiv)

Reference resolution, which aims to identify entities being referred to by a speaker, is more complex in real world settings: new referents may be created by processes the agents engage in and/or be salient only because they belong to the shared physical setting. Our focus is on resolving references to visualizations on a large screen display in multimodal dialogue; crucially, reference resolution is directly involved in the process of creating new visualizations. We describe our annotations for user references to visualizations appearing on a large screen via language and hand gesture and also new entity establishment, which results from executing the user request to create a new visualization. We also describe our reference resolution pipeline which relies on an information-state architecture to maintain dialogue context. We report results on detecting and resolving references, effectiveness of contextual information on the model, and under-specified requests for creating visualizations. We also experiment with conventional CRF and deep learning / transformer models (BiLSTM-CRF and BERT-CRF) for tagging references in user utterance text. Our results show that transfer learning significantly boost performance of the deep learning methods, although CRF still out-performs them, suggesting that conventional methods may generalize better for low resource data.