# Detecting Interlocutor Confusion in Situated Human-Avatar Dialogue: A Pilot Study

arXiv: https://arxiv.org/abs/2206.02436 · HTML: https://mlsanity.com/paper/2206.02436/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 42 references that are in the map.
- First posted on arXiv 6 Jun 2022.

## Builds on (most cited first, 8 of 8)

- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [ETH-XGaze: A Large Scale Dataset for Gaze Estimation under Extreme Head Pose and Gaze Variation](https://mlsanity.com/paper/2007.15837/) (2020)
- [Facial expression and attributes recognition based on multi-task learning of lightweight neural networks](https://mlsanity.com/paper/2103.17107/) (2021)
- [Multi-Modal Emotion recognition on IEMOCAP Dataset using Deep Learning](https://mlsanity.com/paper/1804.05788/) (2018)
- [A Review of Verbal and Non-Verbal Human-Robot Interactive Communication](https://mlsanity.com/paper/1401.4994/) (2014)
- [On-the-fly Detection of User Engagement Decrease in Spontaneous Human-Robot Interaction, International Journal of Social Robotics, 2019](https://mlsanity.com/paper/2004.09596/) (2020)
- [Personality in Healthcare Human Robot Interaction (H-HRI): A Literature Review and Brief Critique](https://mlsanity.com/paper/2008.06723/) (2020)

## Built on by (most cited first, 2 of 2)

- [Dialogue Policies for Confusion Mitigation in Situated HRI](https://mlsanity.com/paper/2208.09367/) (2022)
- [Adapting Robot's Explanation for Failures Based on Observed Human Behavior in Human-Robot Collaboration](https://mlsanity.com/paper/2504.09717/) (2025)

## Abstract (arXiv)

In order to enhance levels of engagement with conversational systems, our long term research goal seeks to monitor the confusion state of a user and adapt dialogue policies in response to such user confusion states. To this end, in this paper, we present our initial research centred on a user-avatar dialogue scenario that we have developed to study the manifestation of confusion and in the long term its mitigation. We present a new definition of confusion that is particularly tailored to the requirements of intelligent conversational system development for task-oriented dialogue. We also present the details of our Wizard-of-Oz based data collection scenario wherein users interacted with a conversational avatar and were presented with stimuli that were in some cases designed to invoke a confused state in the user. Post study analysis of this data is also presented. Here, three pre-trained deep learning models were deployed to estimate base emotion, head pose and eye gaze. Despite a small pilot study group, our analysis demonstrates a significant relationship between these indicators and confusion states. We understand this as a useful step forward in the automated analysis of the pragmatics of dialogue.