# Deep Learning Based Multi-modal Addressee Recognition in Visual Scenes with Utterances

arXiv: https://arxiv.org/abs/1809.04288 · HTML: https://mlsanity.com/paper/1809.04288/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Sep 2018.

## Built on by (most cited first, 3 of 3)

- [Multimodal Conversation Structure Understanding](https://mlsanity.com/paper/2505.17536/) (2025)
- [Real-time Addressee Estimation: Deployment of a Deep-Learning Model on the iCub Robot](https://mlsanity.com/paper/2311.05334/) (2023)
- [On the Structure of Address in Multi-Party Dialogue: From Discrete Labels to Continuous Levels](https://mlsanity.com/paper/2607.15648/) (2026)

## Abstract (arXiv)

With the widespread use of intelligent systems, such as smart speakers, addressee recognition has become a concern in human-computer interaction, as more and more people expect such systems to understand complicated social scenes, including those outdoors, in cafeterias, and hospitals. Because previous studies typically focused only on pre-specified tasks with limited conversational situations such as controlling smart homes, we created a mock dataset called Addressee Recognition in Visual Scenes with Utterances (ARVSU) that contains a vast body of image variations in visual scenes with an annotated utterance and a corresponding addressee for each scenario. We also propose a multi-modal deep-learning-based model that takes different human cues, specifically eye gazes and transcripts of an utterance corpus, into account to predict the conversational addressee from a specific speaker's view in various real-life conversational scenarios. To the best of our knowledge, we are the first to introduce an end-to-end deep learning model that combines vision and transcripts of utterance for addressee recognition. As a result, our study suggests that future addressee recognition can reach the ability to understand human intention in many social situations previously unexplored, and our modality dataset is a first step in promoting research in this field.