# Decision-Theoretic Question Generation for Situated Reference Resolution: An Empirical Study and Computational Model

arXiv: https://arxiv.org/abs/2110.06288 · HTML: https://mlsanity.com/paper/2110.06288/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 51 references that are in the map.
- First posted on arXiv 12 Oct 2021.

## Builds on (most cited first, 6 of 6)

- [Situated and Interactive Multimodal Conversations](https://mlsanity.com/paper/2006.01460/) (2020)
- [A Review of Verbal and Non-Verbal Human-Robot Interactive Communication](https://mlsanity.com/paper/1401.4994/) (2014)
- [Conversation as Action Under Uncertainty](https://mlsanity.com/paper/1301.3883/) (2013)
- [How Should Agents Ask Questions For Situated Learning? An Annotated Dialogue Corpus](https://mlsanity.com/paper/2106.06504/) (2021)
- [Resolving References to Objects in Photographs using the Words-As-Classifiers Model](https://mlsanity.com/paper/1510.02125/) (2015)
- [Model-based Bayesian Reinforcement Learning for Dialogue Management](https://mlsanity.com/paper/1304.1819/) (2013)

## Built on by (most cited first, 3 of 3)

- ['What are you referring to?' Evaluating the Ability of Multi-Modal Dialogue Models to Process Clarificational Exchanges](https://mlsanity.com/paper/2307.15554/) (2023)
- [Repairs in a Block World: A New Benchmark for Handling User Corrections with Multi-Modal Language Models](https://mlsanity.com/paper/2409.14247/) (2024)
- [Frictive Policy Optimization for LLMs: Epistemic Intervention, Risk-Sensitive Control, and Reflective Alignment](https://mlsanity.com/paper/2604.25136/) (2026)

## Abstract (arXiv)

Dialogue agents that interact with humans in situated environments need to manage referential ambiguity across multiple modalities and ask for help as needed. However, it is not clear what kinds of questions such agents should ask nor how the answers to such questions can be used to resolve ambiguity. To address this, we analyzed dialogue data from an interactive study in which participants controlled a virtual robot tasked with organizing a set of tools while engaging in dialogue with a live, remote experimenter. We discovered a number of novel results, including the distribution of question types used to resolve ambiguity and the influence of dialogue-level factors on the reference resolution process. Based on these empirical findings we: (1) developed a computational model for clarification requests using a decision network with an entropy-based utility assignment method that operates across modalities, (2) evaluated the model, showing that it outperforms a slot-filling baseline in environments of varying ambiguity, and (3) interpreted the results to offer insight into the ways that agents can ask questions to facilitate situated reference resolution.